gh-138907: Support RFC 9309 in robotparser

[serhiy-storchaka]

• empty lines are always ignored instead of separating groups
• the "user-agent" line after a rule starts a new group
• groups matching the same user agent are now merged
• the rule with the longest match wins instead of the first matching rule
• in case of equal matches, the “Allow” rule wins over “Disallow”
• special characters “$” and “*” are now supported in rules
• prefer full match for user agent

• Issue: Support RFC 9309 in robotparser #138907

[merwok]

translate* I think