Moreover, it is highly recommended to have an account from an open source end-to-end software development platform with built-in version control like GitLab, Github and so on.
In this documentation, we use GitLab since it has built-in version control, issue tracking, code review, CI/CD, and more.
Installation & Setup
In this section, we describe how to create and set up your first MAGE project.
This section is prepared for Windows. If you using Windows, make sure that The Windows Subsystem for Linux (WSL) is activated. For Linux/MacOS, the instructions are very similar.
This will execute the HelloWorldExperimenter and will generate the following output.
[INFO] Scanning for projects...
[INFO][INFO] -------------------< scop-framework:mage.abmproject >--------------------
[INFO] Building scop-framework.mage.abmproject 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The artifact org.slf4j:slf4j-log4j12:jar:2.0.3 has been relocated to org.slf4j:slf4j-reload4j:jar:2.0.3
[INFO][INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mage.abmproject ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO][INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ mage.abmproject ---
[INFO] Nothing to compile - all classes are up to date
[INFO][INFO] --- exec-maven-plugin:3.1.0:java (default-cli) @ mage.abmproject ---
[INFO] Using the XML loader.
[INFO] Loading configuration
[INFO] Unable to load external configuration file experiments/DefaultExperimenter.xml, maybe it is internal ?
[INFO] Configuration loaded.
[INFO][ 0.00] Running experiment.
[INFO][ 0.00] ThePlatform/Role/ExperimenterAgent is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/ActionScheduler is ACTIVE now.
[INFO][ 0.00] ThePlatform/SchedulerRole/ActionScheduler is ACTIVE now.
[INFO][ 0.00] ThePlatform/ExperimenterRole/ExperimenterAgent is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/HW_Environment is ACTIVE now.
[INFO][ 0.00] HW_Environment/GroupManagerRole/HW_Environment is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/Bob is ACTIVE now.
[INFO][ 0.00] HW_Environment/BystanderRole/Bob is ACTIVE now.
[INFO][ 4.00] Bob says "Hello World !"[INFO][ 9.00] Bob says "Hello World !"[INFO][ 14.00] Bob says "Hello World !"[INFO][ 19.00] Bob says "Hello World !"[INFO][ 24.00] Bob says "Hello World !"[INFO][ 29.00] Bob says "Hello World !"[INFO][ 34.00] Bob says "Hello World !"[INFO][ 39.00] Bob says "Hello World !"[INFO][ 44.00] Bob says "Hello World !"[INFO][ 49.00] Bob says "Hello World !"[INFO][101.00] Done.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.764 s
[INFO] Finished at: 2022-12-15T12:16:55+04:00
[INFO] ------------------------------------------------------------------------
Press any key to continue...
Commit your MAGE project to git
The initial configuration is done. Now you can make your first commit that create the main branch in your git repository.
First, list the untracked files using git status:
> git status
On branch main
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be committed) .gitignore
.gitlab-ci.yml
.m2/
.mage-api-install.sh
.mage-execute.sh
.mage-project-builder.sh
README.md
pom.xml
src/
nothing added to commit but untracked files present (use "git add" to track)>
Then, use git add . to track them all.
> git add .
warning: in the working copy of '.gitignore', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.gitlab-ci.yml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.m2/ci_settings.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.mage-execute.sh', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.mage-project-builder.sh', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'README.md', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'pom.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/java/mage/abmproject/HelloWorldExperimenter.java', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/resources/experiments/DefaultExperimenter.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/resources/log4j.properties', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/test/resources/log4j.properties', LF will be replaced by CRLF the next time Git touches it
Click Window > Show View > Other... and type git to filter options.
Choose Git Repositories from the list and click Open.
Go to the Git Repositories view and click Add an existing local Git Repository to this view.
From the pop-up window Add Git Repositories, find and add mage.abmproject.
From the Git Repositories view, right clich on the mage.abmproject repository and click Import Projects...
Eclipse IDE will automatically find your empty project. Just click Finish and your project will be added in the Project Explorer window.
Commit your MAGE project to git
Now, go back to Eclipse IDE, click on the project root on Project Explorer and press F5 to refresh it.
You will see the updated project content.
Click Window > Show View > Other... and type git to filter options.
Choose Git Staging from the list and click Open.
Go to the Git Staging view, click ++ to add all files to Staged Changes, type "Initial commit." as Commit Message, click Add Signed-off by and finally click Commit and Push.
Congratulations! You have created and configured your MAGE project. Now you can start developing.
Check-out and try the HelloWorld project
Before proceeding, let’s download and run the Hello World example.
Moreover, it is highly recommended to have an account from an open source end-to-end software development platform with built-in version control like GitLab, Github and so on.
In this documentation, we use GitLab since it has built-in version control, issue tracking, code review, CI/CD, and more.
Installation & Setup
In this section, we describe how to create and set up your first SCOP project.
This section is prepared for Windows. If you using Windows, make sure that The Windows Subsystem for Linux (WSL) is activated. For Linux/MacOS, the instructions are very similar.
This will execute the HelloWorldExperimenter and will generate the following output.
[INFO] Scanning for projects...
[INFO][INFO] -------------------< scop-framework:mage.abmproject >--------------------
[INFO] Building scop-framework.mage.abmproject 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The artifact org.slf4j:slf4j-log4j12:jar:2.0.3 has been relocated to org.slf4j:slf4j-reload4j:jar:2.0.3
[INFO][INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mage.abmproject ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO][INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ mage.abmproject ---
[INFO] Nothing to compile - all classes are up to date
[INFO][INFO] --- exec-maven-plugin:3.1.0:java (default-cli) @ mage.abmproject ---
[INFO] Using the XML loader.
[INFO] Loading configuration
[INFO] Unable to load external configuration file experiments/DefaultExperimenter.xml, maybe it is internal ?
[INFO] Configuration loaded.
[INFO][ 0.00] Running experiment.
[INFO][ 0.00] ThePlatform/Role/ExperimenterAgent is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/ActionScheduler is ACTIVE now.
[INFO][ 0.00] ThePlatform/SchedulerRole/ActionScheduler is ACTIVE now.
[INFO][ 0.00] ThePlatform/ExperimenterRole/ExperimenterAgent is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/HW_Environment is ACTIVE now.
[INFO][ 0.00] HW_Environment/GroupManagerRole/HW_Environment is ACTIVE now.
[INFO][ 0.00] ThePlatform/Role/Bob is ACTIVE now.
[INFO][ 0.00] HW_Environment/BystanderRole/Bob is ACTIVE now.
[INFO][ 4.00] Bob says "Hello World !"[INFO][ 9.00] Bob says "Hello World !"[INFO][ 14.00] Bob says "Hello World !"[INFO][ 19.00] Bob says "Hello World !"[INFO][ 24.00] Bob says "Hello World !"[INFO][ 29.00] Bob says "Hello World !"[INFO][ 34.00] Bob says "Hello World !"[INFO][ 39.00] Bob says "Hello World !"[INFO][ 44.00] Bob says "Hello World !"[INFO][ 49.00] Bob says "Hello World !"[INFO][101.00] Done.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.764 s
[INFO] Finished at: 2022-12-15T12:16:55+04:00
[INFO] ------------------------------------------------------------------------
Press any key to continue...
Commit your SCOP project to git
The initial configuration is done. Now you can make your first commit that create the main branch in your git repository.
First, list the untracked files using git status:
> git status
On branch main
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be committed) .gitignore
.gitlab-ci.yml
.m2/
.mage-api-install.sh
.mage-execute.sh
.mage-project-builder.sh
README.md
pom.xml
src/
nothing added to commit but untracked files present (use "git add" to track)>
Then, use git add . to track them all.
> git add .
warning: in the working copy of '.gitignore', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.gitlab-ci.yml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.m2/ci_settings.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.mage-execute.sh', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of '.mage-project-builder.sh', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'README.md', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'pom.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/java/mage/abmproject/HelloWorldExperimenter.java', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/resources/experiments/DefaultExperimenter.xml', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/main/resources/log4j.properties', LF will be replaced by CRLF the next time Git touches it
warning: in the working copy of 'src/test/resources/log4j.properties', LF will be replaced by CRLF the next time Git touches it
Click Window > Show View > Other... and type git to filter options.
Choose Git Repositories from the list and click Open.
Go to the Git Repositories view and click Add an existing local Git Repository to this view.
From the pop-up window Add Git Repositories, find and add mage.abmproject.
From the Git Repositories view, right clich on the mage.abmproject repository and click Import Projects...
Eclipse IDE will automatically find your empty project. Just click Finish and your project will be added in the Project Explorer window.
Commit your SCOP project to git
Now, go back to Eclipse IDE, click on the project root on Project Explorer and press F5 to refresh it.
You will see the updated project content.
Click Window > Show View > Other... and type git to filter options.
Choose Git Staging from the list and click Open.
Go to the Git Staging view, click ++ to add all files to Staged Changes, type "Initial commit." as Commit Message, click Add Signed-off by and finally click Commit and Push.
Congratulations! You have created and configured your SCOP project. Now you can start developing.
Check-out and try the HelloWorld project
Before proceeding, let’s download and run the Hello World example.
An active internet connection to pull base images and dependencies.
Required Files
Ensure that the following file is present in your working directory:
Dockerfile (located in the root directory of your project)
The Dockerfile contains the instructions necessary to build the Docker image that will run your simulation.
Build the Docker Image
To build the Docker image, run the following command in your terminal:
docker build --no-cache -t model-name .
--no-cache: Ensures that the build process does not use any cached layers.
Replace model-name with the desired name for your Docker image.
This process may take some time depending on the complexity of your Dockerfile and the dependencies required.
Run the Docker Image as a Detached Container
After building the image, run it as a detached container using:
docker run -d --name model-name-container model-name
-d: Runs the container in detached mode (in the background).
--name model-name-container: Sets the name of the running container to model-name-container (you can change this to any name you prefer).
model-name: Refers to the Docker image you built earlier.
Verify the Running Container
To verify that your container is running, use:
docker ps
This command lists all active containers. Look for model-name-container in the output to confirm that your container is running.
View Container Logs
To view the logs generated by your running container, use:
docker logs model-name-container
This command helps you monitor the output of your simulation and is useful for debugging any issues.
Stop the Running Container
When you need to stop the container, use:
docker stop model-name-container
This will gracefully stop the container named model-name-container.
Additional Tips
Restart a Stopped Container: If you want to restart a stopped container, use:
docker start model-name-container
Remove a Container: To remove a container after it has stopped, use:
docker rm model-name-container
Remove an Image: If you need to remove the Docker image, use:
docker rmi model-name
By following these steps, you will be able to build, run, monitor, and manage Docker containers for the SCOP Framework simulations. Adjust the commands and options as needed to suit your specific project requirements.
3 - Running on HPC
Guide for running SCOP Framework on HPC systems using SLURM.
This guide covers three scenarios for running the SCOP Framework on HPC systems using SLURM:
Java and Maven are available on the HPC system.
Maven is not available on the HPC system (requiring a locally built fat JAR).
Both Java and Maven are not available on the HPC system (requiring Java installation with EasyBuild).
Checking Available Modules
To verify if the required modules (Java/21.0.2 and Maven/3.9.9) are installed on the HPC, follow these steps:
Check for Java
module avail Java
This command will list all available Java modules. Look for Java/21.0.2 in the output.
Check for Maven
module avail Maven
This command will list all available Maven modules. Look for Maven/3.9.9 in the output.
Case 1: Java and Maven Are Available on the HPC System
In this scenario, both Java and Maven are installed on the HPC environment, allowing you to build and run the SCOP Framework directly on the cluster.
Clone the Repository and Build the Project
Clone the repository of your model and build it using Maven:
git clone https://github.com/your-repo/your-model.git
cd your-model
mvn clean install
This step compiles the Java code and packages the application into a runnable JAR file (typically target/your-model.jar).
Create a SLURM Job Script
Write a job script to run the application, specifying the Java JAR file:
#!/bin/bash
## Set up the job # Note: Adjust the sbatch settings as needed #SBATCH --nodes=8#SBATCH --time=00:05:00#SBATCH --qos=devel#SBATCH --account=account_name# Load Java and Maven modulesmodule --quiet purge # Reset the modules to the system defaultmodule load Java/21.0.2 # Loading Java 21module load Maven/3.9.9 # Loading Maven 3.9.9# Navigate to the your model's directorycd /path/to/your-model
# Run the simulation using the JAR file# Note: Adjust the memory settings (-Xms and -Xmx) as neededjava -Xms1g -Xmx32g -jar target/your-model.jar exec-cli
echo$SLURM_SUBMIT_DIRexit0
Save this script as run_your_model_on_hpc.sh.
Submit the Job
Submit the job using the sbatch command:
sbatch run_your_model_on_hpc.sh
Check the Job
Check the status of the job using the squeue command:
squeue -u your_username
Case 2: Maven Is Not Available on the HPC System
In this scenario, Maven is not available on the HPC, so the build process must be done locally (on a development machine), and then the resulting “fat JAR” is transferred to the HPC system.
Build a Fat JAR Locally
On your local machine (with Maven installed), navigate to the project repository and build a fat JAR that includes all dependencies:
mvn clean compile package -Pshade -DskipTests
This will generate a “fat JAR” file that includes all required libraries and dependencies, typically found in the target folder, e.g., target/your-model-version-fat.jar.
Transfer the Fat JAR to the HPC
Use scp or rsync to copy the fat JAR to the HPC machine:
On the HPC, write a job script to run the fat JAR:
#!/bin/bash
## Set up the job # Note: Adjust the sbatch settings as needed #SBATCH --nodes=8#SBATCH --time=00:05:00#SBATCH --qos=devel#SBATCH --account=account_name# Load Java and Maven modulesmodule --quiet purge # Reset the modules to the system defaultmodule load Java/21.0.2 # Loading Java 21# Navigate to the your model's directorycd /path/to/your-model
# Run the simulation using the JAR file# Note: Adjust the memory settings (-Xms and -Xmx) as neededjava -Xms1g -Xmx32g -jar target/your-model-version-fat.jar exec-cli
echo$SLURM_SUBMIT_DIRexit0
Save this script as run_your_model_on_hpc.sh.
Submit the Job
Submit the job using the sbatch command:
sbatch run_your_model_on_hpc.sh
Check the Job
Check the status of the job using the squeue command:
squeue -u your_username
Case 3: Java and Maven Are Not Available on the HPC System
In this scenario, neither Java nor Maven is available on the HPC system, so Java needs to be installed using EasyBuild, and a fat JAR must be generated locally.
Create an EasyBuild Script to in Java 21
On the HPC, create an EasyBuild script to install Java 21:
easyblock='Binary'name='JDK'version='21'toolchain={'name': 'foss', 'version': '2023b'}# Adjust based on your available toolchains.homepage='https://www.oracle.com/java/'description="Oracle JDK 21 - Java Development Kit"source_urls=['https://download.oracle.com/java/23/latest/']sources=['jdk-21_linux-x64_bin.tar.gz']# Adjust the installation prefix as necessary#install_path = '/cluster/home/ogurcan/software/OracleJDK-23'# Dependencies if neededdependencies=[]# License (if applicable)license_file='LICENCE'# Adjust this to ensure proper extraction and installationinstall_cmd='tar -xzf jdk-21_linux-x64_bin.tar.gz -C %(installdir)s'sanity_check_paths={'files': ['jdk-21.0.1/bin/java', 'jdk-21.0.1/bin/javac'],
'dirs': ['jdk-21.0.1/lib']}moduleclass='lang'
Save this script as JDK-21.eb.
Install Java 21 Using EasyBuild
Use the EasyBuild script to generate the module file:
eb JDK-21.eb --prefix=/path/to/install/directory
Load the Installed Java Module
Load the newly installed Java 21 module:
module use /path/to/install/directory/modules/all
module load Java/21
Add Java to your PATH
Manually add the JDK 23 bin directory to your PATH:
bash export PATH=/path/to/install/directory/JDK/21.0.1/jdk-21.0.1/bin/:$PATH
Verify that Java is installed correctly:
java -version
Continue from Case 2
Now that Java loaded, go to Case 2, and follow the steps.
Conclusion
In all cases, adjust paths and module names according to the specific setup of your HPC environment.
