1. What is systemd?
systemd is the Linux service manager responsible for starting, stopping, restarting and supervising background services.
systemctl is the command-line tool used to control systemd.
systemctl
|
v
systemd
|
+---- apigateway.service
|
+---- auth.service
|
+---- delivery.service
|
+---- governance.service
|
+---- orchestrator.service
|
+---- microservices.target
| Command | Purpose |
|---|---|
systemctl start | Start a service |
systemctl stop | Stop a service |
systemctl restart | Restart a service |
systemctl status | Check service status |
systemctl enable | Enable automatic startup at boot |
systemctl disable | Disable automatic startup |
2. Application Architecture
systemd service grouping
3. Application Directory Structure
/home/srinath/
│
├── opt/
│ ├── openjdk-11.0.27/
│ ├── apache-tomcat-9.0.107/
│ │
│ └── apps/
│ ├── frt-testseries/
│ ├── log/
│ │ ├── apiGateway.*
│ │ ├── authService.*
│ │ ├── deliveryService.*
│ │ ├── governance.*
│ │ └── orchestrator.*
│ │
│ └── autoscripts/
│ └── microservices_systemd/
│ ├── install-microservices-systemd.sh
│ ├── start-*.sh
│ └── *.service
4. Configure Java 11 and Tomcat
Java environment
export JAVA_HOME=/home/srinath/opt/openjdk-11.0.27
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib
Verify:
echo $JAVA_HOME
java -version
which java
Expected Java location:
/home/srinath/opt/openjdk-11.0.27/bin/java
Tomcat environment
export CATALINA_HOME=/home/srinath/opt/apache-tomcat-9.0.107
export PATH=$JAVA_HOME/bin:$CATALINA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$CATALINA_HOME/lib
Verify:
echo $CATALINA_HOME
export JAVA_HOME=... command in your interactive shell does not automatically become the environment of a systemd service. Production service files should explicitly define the required environment or use the full Java executable path.
5. Check Ports Before Starting Services
First check whether API Gateway port 8080 is occupied:
sudo lsof -i :8080
If you find a PID such as:
2521023
First identify the process:
ps -fp 2521023
If it is an unwanted orphaned/manual process and cannot be stopped normally:
sudo kill -9 2521023
Then verify:
sudo lsof -i :8080
Also check the other application ports:
sudo lsof -i :8081
sudo lsof -i :8082
Alternative:
sudo ss -lntp | grep -E ':8080|:8081|:8082'
sudo systemctl stop <service>. Use kill -9 only when a stuck/orphaned process remains.
6. Stop Existing Manually Started Microservices
Before moving to systemd, stop applications that may have been started manually.
cd /home/srinath/opt/apps/frt-testseries/
./stop-microservices.sh
Then check ports again:
sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082
Port already in use.
7. Clean Old Application Logs
cd /home/srinath/opt/apps/log
Review the directory first:
ls -lh
Remove old application log files:
rm -f apiGateway.* deliveryService.* authService.* governance.* orchestrator.*
This is preferable to recursively deleting arbitrary directories when only log files are intended to be removed.
8. Fix Windows Line Endings with dos2unix
If shell scripts were edited on Windows, they may contain CRLF line endings. Linux shell scripts normally require LF line endings.
Install dos2unix
sudo apt-get update
sudo apt-get install -y dos2unix
Convert a script
dos2unix start-artifactory.sh
For the systemd installation script:
dos2unix install-microservices-systemd.sh
Verify
head -n 1 start-artifactory.sh | od -An -t x1
Ensure there is no CR character (0d) at the end of the line.
Make executable
chmod +x start-artifactory.sh
./start-artifactory.sh
/bin/bash^M: bad interpreter, convert the script using dos2unix.
9. Install the systemd Configuration
Go to the systemd scripts directory:
cd /home/srinath/opt/apps/autoscripts/microservices_systemd
Check the files:
ls -lh
Make the installation script executable if necessary:
chmod +x install-microservices-systemd.sh
Run the installation script:
sudo ./install-microservices-systemd.sh
When prompted, select:
1
After installation, check the systemd directory:
ls -lh /etc/systemd/system/
Look for:
auth.service
delivery.service
governance.service
orchestrator.service
apigateway.service
microservices.target
10. Understanding a systemd Service File
A typical Spring Boot service definition looks conceptually like this:
[Unit]
Description=FRT Auth Service
After=network.target
[Service]
Type=simple
User=srinath
WorkingDirectory=/home/srinath/opt/apps/frt-testseries/
Environment="JAVA_HOME=/home/srinath/opt/openjdk-11.0.27"
ExecStart=/home/srinath/opt/openjdk-11.0.27/bin/java \
-jar /home/srinath/opt/apps/frt-testseries/frt-auth-service.jar
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Section meanings
| Section / Property | Purpose |
|---|---|
[Unit] | Service metadata and dependencies. |
After=network.target | Starts the service after the network target is available. |
[Service] | Defines how the application process runs. |
User=srinath | Runs the application as the specified Linux user. |
WorkingDirectory | Sets the application's working directory. |
Environment | Provides environment variables to the service. |
ExecStart | The exact command systemd executes. |
Restart=always | Restarts the application after an unexpected exit. |
RestartSec=10 | Waits 10 seconds before restarting. |
[Install] | Defines how the service participates in boot-time enablement. |
11. Reload systemd Configuration
After installing or modifying service files, reload systemd:
sudo systemctl daemon-reload
This tells systemd to re-read unit files.
12. Manage Individual Microservices
Auth Service
sudo systemctl start auth.service
sudo systemctl stop auth.service
sudo systemctl restart auth.service
sudo systemctl status auth.service
Delivery Service
sudo systemctl start delivery.service
sudo systemctl stop delivery.service
sudo systemctl restart delivery.service
sudo systemctl status delivery.service
Governance Service
sudo systemctl start governance.service
sudo systemctl stop governance.service
sudo systemctl restart governance.service
sudo systemctl status governance.service
Orchestrator Service
sudo systemctl start orchestrator.service
sudo systemctl stop orchestrator.service
sudo systemctl restart orchestrator.service
sudo systemctl status orchestrator.service
API Gateway
sudo systemctl start apigateway.service
sudo systemctl stop apigateway.service
sudo systemctl restart apigateway.service
sudo systemctl status apigateway.service
13. microservices.target
microservices.target can group all application services into one logical unit.
microservices.target
|
+---- auth.service
|
+---- delivery.service
|
+---- governance.service
|
+---- orchestrator.service
|
+---- apigateway.service
Start all
sudo systemctl start microservices.target
Stop all
sudo systemctl stop microservices.target
Restart target
sudo systemctl restart microservices.target
Recommended complete stop/start
sudo systemctl stop microservices.target
sudo systemctl start microservices.target
sudo systemctl status microservices.target
Check target dependencies
systemctl list-dependencies microservices.target
14. Enable Services at Server Boot
Enable Auth
sudo systemctl enable auth.service
sudo systemctl is-enabled auth.service
Expected:
enabled
Disable Auth
sudo systemctl disable auth.service
Enable the complete target
sudo systemctl enable microservices.target
sudo systemctl is-enabled microservices.target
15. Diagnostics and Troubleshooting
Check running services
sudo systemctl --type=service --state=running
Filter application services:
sudo systemctl --type=service | grep -E 'auth|delivery|governance|orchestrator|apigateway'
Check failed services
sudo systemctl --failed
View service logs
sudo journalctl -u auth.service
Last 100 lines
sudo journalctl -u auth.service -n 100 --no-pager
Follow logs live
sudo journalctl -u auth.service -f
For other services:
sudo journalctl -u orchestrator.service -f
sudo journalctl -u apigateway.service -f
sudo journalctl -u delivery.service -f
sudo journalctl -u governance.service -f
Logs since current boot
sudo journalctl -u auth.service -b
Check ports
sudo ss -lntp | grep -E ':8080|:8081|:8082'
Check a service process
ps -ef | grep java
failed, do not immediately keep restarting it. First run systemctl status and journalctl to identify the actual startup error.
16. New Server IP and DNS Verification
Your new server IP is:
103.5.85.49
Check server IP
hostname -I
or:
ip addr
Test API Gateway directly
curl http://103.5.85.49:8080
If Actuator is enabled:
curl http://103.5.85.49:8080/actuator/health
DNS verification
dig apigateway.fortresscareer.com
or:
nslookup apigateway.fortresscareer.com
The DNS A record should ultimately resolve to:
103.5.85.49
17. Nginx + systemd Architecture
If Nginx is the public reverse proxy, API Gateway port 8080 can normally remain an internal application port rather than being directly exposed to the Internet, subject to your firewall/security design.
18. Recommended Production Deployment Procedure
- Configure Java 11.
- Verify Java version and path.
- Go to the systemd scripts directory.
- Check ports 8080, 8081 and 8082.
- Stop manually started microservices.
- Verify that old processes have stopped.
- Clean old application logs if required.
- Convert scripts from CRLF to LF using
dos2unixif needed. - Make scripts executable.
- Run
install-microservices-systemd.sh. - Select option
1. - Run
systemctl daemon-reload. - Verify all generated service files.
- Start
microservices.target. - Check the target status.
- Check each individual service.
- Check listening ports.
- Check systemd failures.
- Check application logs.
- Test the application through the server IP.
- Verify DNS resolution.
- Verify Nginx and HTTPS.
Copy/Paste Production Sequence
# 1. Java
export JAVA_HOME=/home/srinath/opt/openjdk-11.0.27
export PATH=$JAVA_HOME/bin:$PATH
java -version
# 2. Stop manually started applications
cd /home/srinath/opt/apps/frt-testseries/
./stop-microservices.sh
# 3. Check ports
sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082
# 4. Clean logs if required
cd /home/srinath/opt/apps/log
rm -f apiGateway.* deliveryService.* authService.* governance.* orchestrator.*
# 5. Install systemd configuration
cd /home/srinath/opt/apps/autoscripts/microservices_systemd
chmod +x install-microservices-systemd.sh
sudo ./install-microservices-systemd.sh
# Select option 1
# 6. Reload systemd
sudo systemctl daemon-reload
# 7. Start all services
sudo systemctl start microservices.target
# 8. Check target
sudo systemctl status microservices.target
# 9. Check services
sudo systemctl status apigateway.service
sudo systemctl status auth.service
sudo systemctl status delivery.service
sudo systemctl status governance.service
sudo systemctl status orchestrator.service
# 10. Check ports
sudo ss -lntp | grep -E ':8080|:8081|:8082'
# 11. Check failures
sudo systemctl --failed
# 12. Check logs
sudo journalctl -u apigateway.service -n 100 --no-pager
sudo journalctl -u orchestrator.service -n 100 --no-pager
sudo journalctl -u auth.service -n 100 --no-pager
19. Emergency Restart Procedure
Use this when the complete application stack is stuck or requires a controlled restart.
# Stop all application services
sudo systemctl stop microservices.target
# Check for failed services
sudo systemctl --failed
# Check important ports
sudo lsof -i :8080
sudo lsof -i :8081
sudo lsof -i :8082
If an orphaned process remains:
ps -fp <PID>
Only if it is confirmed to be an unwanted stuck/orphaned process:
sudo kill -9 <PID>
Then restart:
sudo systemctl start microservices.target
sudo systemctl status microservices.target
Verify each service:
sudo systemctl status apigateway.service
sudo systemctl status auth.service
sudo systemctl status delivery.service
sudo systemctl status governance.service
sudo systemctl status orchestrator.service
20. systemctl Cheat Sheet
| Requirement | Command |
|---|---|
| Start | sudo systemctl start auth.service |
| Stop | sudo systemctl stop auth.service |
| Restart | sudo systemctl restart auth.service |
| Status | sudo systemctl status auth.service |
| Enable at boot | sudo systemctl enable auth.service |
| Disable at boot | sudo systemctl disable auth.service |
| Is enabled? | sudo systemctl is-enabled auth.service |
| Is active? | sudo systemctl is-active auth.service |
| Reload configuration | sudo systemctl daemon-reload |
| Failed services | sudo systemctl --failed |
| Service logs | sudo journalctl -u auth.service |
| Last 100 logs | sudo journalctl -u auth.service -n 100 |
| Live logs | sudo journalctl -u auth.service -f |
| Boot logs | sudo journalctl -u auth.service -b |
| Target dependencies | systemctl list-dependencies microservices.target |
21. Final Production Checklist
- Java 11 is installed and verified.
JAVA_HOMEpoints to/home/srinath/opt/openjdk-11.0.27.- Tomcat path is
/home/srinath/opt/apache-tomcat-9.0.107where required. - Server IP is verified as
103.5.85.49. - Ports 8080, 8081 and 8082 are checked before startup.
- Manually running application processes are stopped.
- Systemd installation script is executable.
- Windows CRLF line endings are removed from shell scripts.
install-microservices-systemd.shcompletes successfully.- Expected service files exist under
/etc/systemd/system/. sudo systemctl daemon-reloadis executed after changes.microservices.targetdependencies are verified.- API Gateway is running.
- Auth service is running.
- Delivery service is running.
- Governance service is running.
- Orchestrator service is running.
- Required services are enabled for server boot.
- No services appear under
systemctl --failed. - Application ports are listening as expected.
- systemd and application logs have been checked.
- DNS points to
103.5.85.49. - Nginx reverse proxy configuration points to the correct local service.
- HTTPS and application health endpoints are tested.
Server Boot → systemd → microservices.target → individual services → Java/Spring Boot → API Gateway → Nginx → HTTPS → Users
Important Operational Notes
- Prefer
systemctl stopoverkill -9for systemd-managed processes. - Always run
systemctl daemon-reloadafter changing unit files. - Use
systemctl statusfirst, thenjournalctlwhen diagnosing startup failures. - Do not rely on interactive shell exports for systemd services; define required environment values in the unit configuration.
- Keep API Gateway, Orchestrator and downstream service ports consistent with the actual Spring Boot configuration.
- After a server IP change, verify DNS, firewall/security groups, Nginx configuration and SSL certificates as applicable.
- Keep the systemd service definitions and installation script under version control where possible.