RADICAL-Pilot 0.23 Documentation

RADICAL-Pilot is a Pilot Job system written in Python. It allows a user to run large numbers of computational tasks (called ComputeUnits) concurrently on one or more remote ComputePilots that RADICAL-Pilot can start transparently on a multitude of different distributed resources, like HPC clusters and Clouds.

In this model, the resource is acquired by a user’s application so that the application can schedule ComputeUnits into that resource directly, rather than going through the system’s job scheduler. In many cases, this can drastically shorten overall exeuction time as the individual ComputeUnits don’t have to wait in the system’s scheduler queue but can execute directly on the ComputePilots.

Mailing Lists

• For users: https://groups.google.com/d/forum/radical-pilot-users
• For developers: https://groups.google.com/d/forum/radical-pilot-devel

Contents:

• 1. Introduction
• 2. Installation
  • 2.1. Requirements
  • 2.2. Installation
  • 2.3. Troubleshooting
• 3. Using Local and Remote HPC Resources
  • 3.1. Introduction
  • 3.2. Configuring SSH Access
  • 3.3. Pre-Configured Resources
  • 3.4. Writing a Custom Resource Configuration File
  • 3.5. Customizing Resource Configurations Programatically
• 4. Data Staging
  • 4.1. Compute Unit I/O
  • 4.2. Compute Pilot I/O
  • 4.3. Examples
• 5. Examples & Best Practice
  • 5.1. Getting Started
  • 5.2. Error Handling Strategies
  • 5.3. Disconnecting and Reconnecting
  • 5.4. Launching Remote / HPC ComputePilots
  • 5.5. Executing Multiple Commands in a Single ComputeUnit
  • 5.6. Executing Multicore / Multithreaded ComputeUnits
  • 5.7. Application Control Flow with Callbacks
  • 5.8. Mandelbrot set
  • 5.9. K-Means Clustering
• 6. Tutorial
  • 6.1. Simple Bag-of-Tasks
  • 6.2. Chained Tasks
  • 6.3. Coupled Tasks
  • 6.4. MPI tasks
• 7. Unit Scheduler
  • 7.1. Introduction
  • 7.2. Direct Submission Scheduler (SCHED_DIRECT_SUBMISSION)
  • 7.3. Round Robin Scheduler (SCHED_ROUND_ROBIN)
• 8. Testing
  • 8.1. Introduction
  • 8.2. Remote Testing
  • 8.3. Adding New Tests
• 9. Benchmarks
• 10. Frequently Asked Questions
  • 10.1. Q: I see the error “OperationFailure: too many namespaces/collections”
  • 10.2. A: Try Cleaning-up MongoDB.
  • 10.3. Q: I see the error “Permission denied (publickey,keyboard-interactive).” in AGENT.STDERR or STDERR.
  • 10.4. A: Set up password-less, intra-node SSH access.
  • 10.5. Q: On Gordon I see “Failed to execvp() ‘mybinary’: No such file or directory (2)”
  • 10.6. A: You need to specify the full path of the executable as mpirun_rsh is not able to find it in the path
  • 10.7. Q: How many concurrent RADICAL-Pilot scripts can I execute?
  • 10.8. Q: Why do I get errors from setuptools when trying to use a virtualenv?
  • 10.9. A: Most likely because an upgrade of pip or setuptools failed
• 11. Developer Documentation
  • 11.1. Installation from Source
  • 11.2. License
  • 11.3. Style Guide
  • 11.4. Debugging
  • 11.5. RADICAL-Pilot Architecture
• 12. API Reference
  • 12.1. Sessions and Security Contexts
  • 12.2. Pilots and PilotManagers
  • 12.3. ComputeUnits and UnitManagers
  • 12.4. Exceptions
  • 12.5. State Models
• 13. Release Notes
  • 13.1. 0.23 Release 2014-12-13
  • 13.2. 0.22 Release 2014-12-11
  • 13.3. 0.21 Release 2014-10-29
  • 13.4. 0.19 Release September 12. 2014
  • 13.5. 0.18 Release July 22. 2014
  • 13.6. 0.17 Release June 18. 2014
  • 13.7. 0.16 Release June 17. 2014
  • 13.8. 0.15 Release June 12. 2014
  • 13.9. 0.14 Release June 11. 2014
  • 13.10. 0.13 Release May 19. 2014
  • 13.11. 0.12 Release May 09. 2014
  • 13.12. 0.11 Release Apr. 29. 2014
  • 13.13. 0.10 Release Apr. 29. 2014
  • 13.14. 0.9 Release Apr. 16. 2014
  • 13.15. 0.8 Release Mar. 24. 2014
  • 13.16. 0.7 Release Feb. 25. 2014
  • 13.17. 0.6 Release Feb. 24. 2014
  • 13.18. 0.5 Release Feb. 06. 2014
  • 13.19. 0.4 Release
  • 13.20. 0.1.3 Release
  • 13.21. 0.1.2 Release