Add S3 storage and multi-hop transfer tutorial section#766
Open
alessio94 wants to merge 3 commits intorucio:mainfrom
Open
Add S3 storage and multi-hop transfer tutorial section#766alessio94 wants to merge 3 commits intorucio:mainfrom
alessio94 wants to merge 3 commits intorucio:mainfrom
Conversation
Addresses feedback from PR rucio#744: - Remove Jupyter notebook - Inline script content directly as code blocks - Replace inline script comments with prose descriptions - Remove docker exec wrapping; assume reader is in a Rucio admin environment - Narrow scope to S3 RSE setup only; remove environment initialization steps
for more information, see https://pre-commit.ci
voetberg
reviewed
Mar 26, 2026
|
|
||
| This tutorial covers how to register S3-compatible storage (MinIO) as Rucio Storage Elements (RSEs), configure credentials for both Rucio and FTS, and set up RSE distances to enable multi-hop transfers between S3 and XRootD endpoints. | ||
|
|
||
| The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already operating within a Rucio admin environment with the `rucio` and `rucio-admin` CLI tools available. |
Contributor
There was a problem hiding this comment.
For clarity, rucio and rucio-admin have been merged since rucio 38, the admin role is handled by the permission policies. Instead I'd change this too
Suggested change
| The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already operating within a Rucio admin environment with the `rucio` and `rucio-admin` CLI tools available. | |
| The examples use a Docker Compose playground environment with two MinIO instances (MINIO1, MINIO2) and three XRootD servers (XRD1, XRD2, XRD3). The commands assume you are already have an rucio instance with an admin account. |
voetberg
reviewed
Mar 26, 2026
|
|
||
| Register both MinIO instances as RSEs with S3 protocol configuration. The `gfal.NoRename` implementation is used because S3 does not support server-side rename operations. | ||
|
|
||
| ```bash |
Contributor
There was a problem hiding this comment.
This can be reduced to a for loop (over both MINIO1 and 2), it would make this easier to read
voetberg
reviewed
Mar 26, 2026
| --prefix /rucio/ \ | ||
| --impl rucio.rse.protocols.gfal.NoRename \ | ||
| --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}' | ||
| rucio rse attribute add MINIO1 --key sign_url --value s3 |
Contributor
There was a problem hiding this comment.
These could use some inline comments to explain each of the options, or at least a link to the config params page in the section description
voetberg
reviewed
Mar 26, 2026
| ```bash | ||
| ID1=$(rucio rse show MINIO1 | grep '^ id:' | awk '{print$2}') | ||
| ID2=$(rucio rse show MINIO2 | grep '^ id:' | awk '{print$2}') | ||
| cat >/opt/rucio/etc/rse-accounts.cfg <<JSON |
Contributor
There was a problem hiding this comment.
This doesn't render to anything special, I'd instead just use the console tag to instead do
$ cat /opt/rucio/etc/rse-accounts.cfg
{
"$ID1": {
"access_key": "admin",
"secret_key": "password",
"signature_version": "s3v4",
"region": "us-east-1"
},
"$ID2": {
"access_key": "admin",
"secret_key": "password",
"signature_version": "s3v4",
"region": "us-east-1"
}
}
voetberg
reviewed
Mar 26, 2026
|
|
||
| ### Configuring RSE Distances for Multi-Hop | ||
|
|
||
| RSE distances tell Rucio which transfer paths are available and their relative cost. Setting a distance of 1 between MINIO RSEs and XRD3 establishes the multi-hop path: transfers from MinIO to XRD1 or XRD2 will route through XRD3 as an intermediate. |
Contributor
There was a problem hiding this comment.
This explanation is technically correct, but a little overly verbose. I recommend putting in a mermaid chart instead to make this more visual.
Suggested change
| RSE distances tell Rucio which transfer paths are available and their relative cost. Setting a distance of 1 between MINIO RSEs and XRD3 establishes the multi-hop path: transfers from MinIO to XRD1 or XRD2 will route through XRD3 as an intermediate. | |
| RSE distances establish transfer paths between RSEs. Setting a distance of 1 between the source and an intermediate will ensure the intermediate transfer will always be preferred over longer direct transfers. | |
| ```mermaid | |
| graph TD | |
| MINIO[MINIO RSE] | |
| XRD1[XRD1 RSE] | |
| XRD2[XRD2 RSE] | |
| XRD3[XRD3 RSE] | |
| MINIO -.->|distance=1| XRD3 | |
| XRD3 -.->|distance=1| XRD1 | |
| XRD3 -.->|distance=1| XRD2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supersedes #744 and #739.
Addresses feedback from PR #744:
@voetberg