FRT Test Series – Systemd Configuration & Operations Runbook

Step-by-step Linux systemd configuration, deployment, startup, shutdown, troubleshooting and production operations guide
Server IP: 103.5.85.49 Java: OpenJDK 11.0.27 Tomcat: 9.0.107 API Gateway: 8080 Orchestrator: 8081 Auth: 8082

1. What is systemd?

systemd is the Linux service manager responsible for starting, stopping, restarting and supervising background services.

systemctl is the command-line tool used to control systemd.

systemctl
   |
   v
 systemd
   |
   +---- apigateway.service
   |
   +---- auth.service
   |
   +---- delivery.service
   |
   +---- governance.service
   |
   +---- orchestrator.service
   |
   +---- microservices.target
CommandPurpose
systemctl startStart a service
systemctl stopStop a service
systemctl restartRestart a service
systemctl statusCheck service status
systemctl enableEnable automatic startup at boot
systemctl disableDisable automatic startup

2. Application Architecture

Internet | | HTTPS v 103.5.85.49 | v Nginx :443 | | proxy_pass v API Gateway :8080 | v Orchestrator :8081 | +---- Auth :8082 | +---- Delivery Service | +---- Governance Service

systemd service grouping

microservices.target | +---------------+---------------+ | | | v v v auth.service orchestrator.service apigateway.service | | | 8082 8081 8080 +---------------+---------------+ | | v v delivery.service governance.service

3. Application Directory Structure

/home/srinath/
│
├── opt/
│   ├── openjdk-11.0.27/
│   ├── apache-tomcat-9.0.107/
│   │
│   └── apps/
│       ├── frt-testseries/
│       ├── log/
│       │   ├── apiGateway.*
│       │   ├── authService.*
│       │   ├── deliveryService.*
│       │   ├── governance.*
│       │   └── orchestrator.*
│       │
│       └── autoscripts/
│           └── microservices_systemd/
│               ├── install-microservices-systemd.sh
│               ├── start-*.sh
│               └── *.service

4. Configure Java 11 and Tomcat

Java environment

export JAVA_HOME=/home/srinath/opt/openjdk-11.0.27
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib

Verify:

echo $JAVA_HOME
java -version
which java

Expected Java location:

/home/srinath/opt/openjdk-11.0.27/bin/java

Tomcat environment

export CATALINA_HOME=/home/srinath/opt/apache-tomcat-9.0.107

export PATH=$JAVA_HOME/bin:$CATALINA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$CATALINA_HOME/lib

Verify:

echo $CATALINA_HOME
Important: An export JAVA_HOME=... command in your interactive shell does not automatically become the environment of a systemd service. Production service files should explicitly define the required environment or use the full Java executable path.

5. Check Ports Before Starting Services

First check whether API Gateway port 8080 is occupied:

sudo lsof -i :8080

If you find a PID such as:

2521023

First identify the process:

ps -fp 2521023

If it is an unwanted orphaned/manual process and cannot be stopped normally:

sudo kill -9 2521023

Then verify:

sudo lsof -i :8080

Also check the other application ports:

sudo lsof -i :8081
sudo lsof -i :8082

Alternative:

sudo ss -lntp | grep -E ':8080|:8081|:8082'
Do not use kill -9 as the first option for a systemd-managed service. If systemd owns the process, use sudo systemctl stop <service>. Use kill -9 only when a stuck/orphaned process remains.

6. Stop Existing Manually Started Microservices

Before moving to systemd, stop applications that may have been started manually.

cd /home/srinath/opt/apps/frt-testseries/
./stop-microservices.sh

Then check ports again:

sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082
Why? Running the same Spring Boot application manually and through systemd at the same time can cause errors such as Port already in use.

7. Clean Old Application Logs

cd /home/srinath/opt/apps/log

Review the directory first:

ls -lh

Remove old application log files:

rm -f apiGateway.* deliveryService.* authService.* governance.* orchestrator.*

This is preferable to recursively deleting arbitrary directories when only log files are intended to be removed.

8. Fix Windows Line Endings with dos2unix

If shell scripts were edited on Windows, they may contain CRLF line endings. Linux shell scripts normally require LF line endings.

Install dos2unix

sudo apt-get update
sudo apt-get install -y dos2unix

Convert a script

dos2unix start-artifactory.sh

For the systemd installation script:

dos2unix install-microservices-systemd.sh

Verify

head -n 1 start-artifactory.sh | od -An -t x1

Ensure there is no CR character (0d) at the end of the line.

Make executable

chmod +x start-artifactory.sh
./start-artifactory.sh
If you see errors such as /bin/bash^M: bad interpreter, convert the script using dos2unix.

9. Install the systemd Configuration

Go to the systemd scripts directory:

cd /home/srinath/opt/apps/autoscripts/microservices_systemd

Check the files:

ls -lh

Make the installation script executable if necessary:

chmod +x install-microservices-systemd.sh

Run the installation script:

sudo ./install-microservices-systemd.sh

When prompted, select:

1

After installation, check the systemd directory:

ls -lh /etc/systemd/system/

Look for:

auth.service
delivery.service
governance.service
orchestrator.service
apigateway.service
microservices.target

10. Understanding a systemd Service File

A typical Spring Boot service definition looks conceptually like this:

[Unit]
Description=FRT Auth Service
After=network.target

[Service]
Type=simple
User=srinath
WorkingDirectory=/home/srinath/opt/apps/frt-testseries/

Environment="JAVA_HOME=/home/srinath/opt/openjdk-11.0.27"

ExecStart=/home/srinath/opt/openjdk-11.0.27/bin/java \
  -jar /home/srinath/opt/apps/frt-testseries/frt-auth-service.jar

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Section meanings

Section / PropertyPurpose
[Unit]Service metadata and dependencies.
After=network.targetStarts the service after the network target is available.
[Service]Defines how the application process runs.
User=srinathRuns the application as the specified Linux user.
WorkingDirectorySets the application's working directory.
EnvironmentProvides environment variables to the service.
ExecStartThe exact command systemd executes.
Restart=alwaysRestarts the application after an unexpected exit.
RestartSec=10Waits 10 seconds before restarting.
[Install]Defines how the service participates in boot-time enablement.
systemctl start auth.service | v systemd | v reads auth.service | v ExecStart | v Java 11 | v Spring Boot Auth | v Port 8082

11. Reload systemd Configuration

After installing or modifying service files, reload systemd:

sudo systemctl daemon-reload

This tells systemd to re-read unit files.

12. Manage Individual Microservices

Auth Service

sudo systemctl start auth.service
sudo systemctl stop auth.service
sudo systemctl restart auth.service
sudo systemctl status auth.service

Delivery Service

sudo systemctl start delivery.service
sudo systemctl stop delivery.service
sudo systemctl restart delivery.service
sudo systemctl status delivery.service

Governance Service

sudo systemctl start governance.service
sudo systemctl stop governance.service
sudo systemctl restart governance.service
sudo systemctl status governance.service

Orchestrator Service

sudo systemctl start orchestrator.service
sudo systemctl stop orchestrator.service
sudo systemctl restart orchestrator.service
sudo systemctl status orchestrator.service

API Gateway

sudo systemctl start apigateway.service
sudo systemctl stop apigateway.service
sudo systemctl restart apigateway.service
sudo systemctl status apigateway.service

13. microservices.target

microservices.target can group all application services into one logical unit.

microservices.target
       |
       +---- auth.service
       |
       +---- delivery.service
       |
       +---- governance.service
       |
       +---- orchestrator.service
       |
       +---- apigateway.service

Start all

sudo systemctl start microservices.target

Stop all

sudo systemctl stop microservices.target

Restart target

sudo systemctl restart microservices.target
A target is primarily a grouping/dependency mechanism. Depending on the dependencies configured in your unit files, restarting a target does not necessarily behave exactly like explicitly restarting every service. For a guaranteed application restart, use the stop/start sequence or restart each service explicitly.

Recommended complete stop/start

sudo systemctl stop microservices.target
sudo systemctl start microservices.target
sudo systemctl status microservices.target

Check target dependencies

systemctl list-dependencies microservices.target

14. Enable Services at Server Boot

Enable Auth

sudo systemctl enable auth.service
sudo systemctl is-enabled auth.service

Expected:

enabled

Disable Auth

sudo systemctl disable auth.service

Enable the complete target

sudo systemctl enable microservices.target
sudo systemctl is-enabled microservices.target
Linux server reboot | v systemd | v microservices.target | +---- Auth +---- Delivery +---- Governance +---- Orchestrator +---- API Gateway

15. Diagnostics and Troubleshooting

Check running services

sudo systemctl --type=service --state=running

Filter application services:

sudo systemctl --type=service | grep -E 'auth|delivery|governance|orchestrator|apigateway'

Check failed services

sudo systemctl --failed

View service logs

sudo journalctl -u auth.service

Last 100 lines

sudo journalctl -u auth.service -n 100 --no-pager

Follow logs live

sudo journalctl -u auth.service -f

For other services:

sudo journalctl -u orchestrator.service -f
sudo journalctl -u apigateway.service -f
sudo journalctl -u delivery.service -f
sudo journalctl -u governance.service -f

Logs since current boot

sudo journalctl -u auth.service -b

Check ports

sudo ss -lntp | grep -E ':8080|:8081|:8082'

Check a service process

ps -ef | grep java
If a service is failed, do not immediately keep restarting it. First run systemctl status and journalctl to identify the actual startup error.

16. New Server IP and DNS Verification

Your new server IP is:

103.5.85.49

Check server IP

hostname -I
or:
ip addr

Test API Gateway directly

curl http://103.5.85.49:8080

If Actuator is enabled:

curl http://103.5.85.49:8080/actuator/health

DNS verification

dig apigateway.fortresscareer.com
or:
nslookup apigateway.fortresscareer.com

The DNS A record should ultimately resolve to:

103.5.85.49
apigateway.fortresscareer.com | | DNS A Record v 103.5.85.49 | v Nginx | v API Gateway :8080

17. Nginx + systemd Architecture

INTERNET | | HTTPS :443 v +-------------+ | Nginx | | 103.5.85.49 | +------+------+ | | proxy_pass v +-------------+ | API Gateway | | :8080 | +------+------+ | v +-------------+ |Orchestrator | | :8081 | +------+------+ | +---------+---------+ | | | v v v Auth Delivery Governance :8082

If Nginx is the public reverse proxy, API Gateway port 8080 can normally remain an internal application port rather than being directly exposed to the Internet, subject to your firewall/security design.

18. Recommended Production Deployment Procedure

  1. Configure Java 11.
  2. Verify Java version and path.
  3. Go to the systemd scripts directory.
  4. Check ports 8080, 8081 and 8082.
  5. Stop manually started microservices.
  6. Verify that old processes have stopped.
  7. Clean old application logs if required.
  8. Convert scripts from CRLF to LF using dos2unix if needed.
  9. Make scripts executable.
  10. Run install-microservices-systemd.sh.
  11. Select option 1.
  12. Run systemctl daemon-reload.
  13. Verify all generated service files.
  14. Start microservices.target.
  15. Check the target status.
  16. Check each individual service.
  17. Check listening ports.
  18. Check systemd failures.
  19. Check application logs.
  20. Test the application through the server IP.
  21. Verify DNS resolution.
  22. Verify Nginx and HTTPS.

Copy/Paste Production Sequence

# 1. Java
export JAVA_HOME=/home/srinath/opt/openjdk-11.0.27
export PATH=$JAVA_HOME/bin:$PATH
java -version

# 2. Stop manually started applications
cd /home/srinath/opt/apps/frt-testseries/
./stop-microservices.sh

# 3. Check ports
sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082

# 4. Clean logs if required
cd /home/srinath/opt/apps/log
rm -f apiGateway.* deliveryService.* authService.* governance.* orchestrator.*

# 5. Install systemd configuration
cd /home/srinath/opt/apps/autoscripts/microservices_systemd
chmod +x install-microservices-systemd.sh
sudo ./install-microservices-systemd.sh

# Select option 1

# 6. Reload systemd
sudo systemctl daemon-reload

# 7. Start all services
sudo systemctl start microservices.target

# 8. Check target
sudo systemctl status microservices.target

# 9. Check services
sudo systemctl status apigateway.service
sudo systemctl status auth.service
sudo systemctl status delivery.service
sudo systemctl status governance.service
sudo systemctl status orchestrator.service

# 10. Check ports
sudo ss -lntp | grep -E ':8080|:8081|:8082'

# 11. Check failures
sudo systemctl --failed

# 12. Check logs
sudo journalctl -u apigateway.service -n 100 --no-pager
sudo journalctl -u orchestrator.service -n 100 --no-pager
sudo journalctl -u auth.service -n 100 --no-pager

19. Emergency Restart Procedure

Use this when the complete application stack is stuck or requires a controlled restart.

# Stop all application services
sudo systemctl stop microservices.target

# Check for failed services
sudo systemctl --failed

# Check important ports
sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082

If an orphaned process remains:

ps -fp <PID>

Only if it is confirmed to be an unwanted stuck/orphaned process:

sudo kill -9 <PID>

Then restart:

sudo systemctl start microservices.target
sudo systemctl status microservices.target

Verify each service:

sudo systemctl status apigateway.service
sudo systemctl status auth.service
sudo systemctl status delivery.service
sudo systemctl status governance.service
sudo systemctl status orchestrator.service

20. systemctl Cheat Sheet

RequirementCommand
Startsudo systemctl start auth.service
Stopsudo systemctl stop auth.service
Restartsudo systemctl restart auth.service
Statussudo systemctl status auth.service
Enable at bootsudo systemctl enable auth.service
Disable at bootsudo systemctl disable auth.service
Is enabled?sudo systemctl is-enabled auth.service
Is active?sudo systemctl is-active auth.service
Reload configurationsudo systemctl daemon-reload
Failed servicessudo systemctl --failed
Service logssudo journalctl -u auth.service
Last 100 logssudo journalctl -u auth.service -n 100
Live logssudo journalctl -u auth.service -f
Boot logssudo journalctl -u auth.service -b
Target dependenciessystemctl list-dependencies microservices.target

21. Final Production Checklist

  • Java 11 is installed and verified.
  • JAVA_HOME points to /home/srinath/opt/openjdk-11.0.27.
  • Tomcat path is /home/srinath/opt/apache-tomcat-9.0.107 where required.
  • Server IP is verified as 103.5.85.49.
  • Ports 8080, 8081 and 8082 are checked before startup.
  • Manually running application processes are stopped.
  • Systemd installation script is executable.
  • Windows CRLF line endings are removed from shell scripts.
  • install-microservices-systemd.sh completes successfully.
  • Expected service files exist under /etc/systemd/system/.
  • sudo systemctl daemon-reload is executed after changes.
  • microservices.target dependencies are verified.
  • API Gateway is running.
  • Auth service is running.
  • Delivery service is running.
  • Governance service is running.
  • Orchestrator service is running.
  • Required services are enabled for server boot.
  • No services appear under systemctl --failed.
  • Application ports are listening as expected.
  • systemd and application logs have been checked.
  • DNS points to 103.5.85.49.
  • Nginx reverse proxy configuration points to the correct local service.
  • HTTPS and application health endpoints are tested.
Final expected flow:
Server Boot → systemd → microservices.target → individual services → Java/Spring Boot → API Gateway → Nginx → HTTPS → Users

Important Operational Notes

  • Prefer systemctl stop over kill -9 for systemd-managed processes.
  • Always run systemctl daemon-reload after changing unit files.
  • Use systemctl status first, then journalctl when diagnosing startup failures.
  • Do not rely on interactive shell exports for systemd services; define required environment values in the unit configuration.
  • Keep API Gateway, Orchestrator and downstream service ports consistent with the actual Spring Boot configuration.
  • After a server IP change, verify DNS, firewall/security groups, Nginx configuration and SSL certificates as applicable.
  • Keep the systemd service definitions and installation script under version control where possible.