A new "pants next-gen" command-line parser. #22808

benjyw · 2025-10-26T17:56:27Z

This supports the command/subcommand paradigm, rather than
the "list of goals" paradigm of the existing cli parser.

This is experimental, not wired up to anything yet, and intended to
support a hypothetical "next-generation Pants" (pants_ng for short).

It is not intended to replicate all current Pants CLI nuances, but is a
much simpler design based on our experience with the complexities
of the current CLI parser.

In particular, this new CLI parser does not need to know in advance
about any registered goals and options. The old parser needed this
to tell the difference between specs and goals, and this added a
lot of complexity for little gain. This was also ambiguous, since
introduction of a new goal could change the meaning of an existing
invocation.

This parser instead requires specs to contain a filepath separator,
anything else is a flag or a command or a subcommand, and those
are easily distinguished by syntax alone. A spec path in the root dir
can be prefixed with ./ to meet this requirement.

This parser supports multiple cmd/subcommand pairs, each with
flags, by separating them with a standalone + character. This,
again, removes ambiguity.

This parser supports only long flags (prefixed by --). We currently
support a handful of short flags (such as -l), but those cause
ambiguity since we also use a single dash prefix for negating
specs.

See #22692 for context.

This supports the command/subcommand paradigm, rather than the "list of goals" paradigm of the existing cli parser. This is experimental, not wired up to anything yet, and intended to support a "next-generation Pants" (pants_ng for short). It is not intended to support all current Pants 2 use-cases, but is a much simpler design based on our experience with the complexities of the current cli parser. In particular, this new cli parser does not need to know about any registered options (unlike the old one, which uses them to tell the difference between specs and goals). This parser instead requires specs to contain a filepath separator. This cli parser currently supports one cmd (and optionally one subcommand) per invocation. If we want to support multiple cmd/subcommand pairs in a single invocation in the future we can do so by using a delimiter that isn't interpreted by the shell. See pantsbuild#22692 for context.

sureshjoshi · 2025-10-26T22:45:34Z

Haven't reviewed this yet (trying to get that stupid JS update landed first), but one idea would be to plan for bash/zsh completions early on. I think several of the clap/clap-derivatives have that built-in.

Better than trying to tack it on.

The one we have in mainline, that I intend to update in a few PRs, is finally "good enough" because of how fast the call-by-name made running everything. Previously, it was brutal to have to wait.

However, nothing will beat auto-generated out of the box (not necessarily this PR though)

benjyw · 2025-10-26T23:44:01Z

Haven't reviewed this yet (trying to get that stupid JS update landed first), but one idea would be to plan for bash/zsh completions early on. I think several of the clap/clap-derivatives have that built-in.

Better than trying to tack it on.

The one we have in mainline, that I intend to update in a few PRs, is finally "good enough" because of how fast the call-by-name made running everything. Previously, it was brutal to have to wait.

However, nothing will beat auto-generated out of the box (not necessarily this PR though)

To be clear, this just parses CLIs, but has no knowledge at all of what commands or flags are available. That would be a separate thing, which (unlike this parser) is aware of every backend. And which I agree should plan for completions.

cognifloyd

I'm bike shedding here. I don't intend to nitpick or be mean or whatever. Just thinking through what it would be like to use this.

Thank you for working on this. And _ng seems like a fine non-committal suffix.

cognifloyd · 2025-10-26T19:45:37Z

src/rust/pants_ng/options/src/pants_invocation.rs

+//   cmd1 --cmd1-flags (subcmd1 --subcmd1-flags)? \
+//   @ cmd2 --cmd2-flags (subcmd2 --subcmd2-flags)? \
+//   ... \
+//   @ cmdN --cmdN-flags (subcmdN --subcmdN-flags)? \


I might be odd here, but I like to "read" the commands I type replacing symbols and single characters with a word or functional description. When I write s/this/that/ in a sed command, instead of "s" I mentally say "switch this with that".

So, the @ reads oddly to me: "pants cmd1 ... (AT) cmd2 ..." sounds like cmd1 is a subcmd of cmd2. I'm not saying I want something different, but I would like to understand your thinking better here.

Other potential separators:

+
like "and" without using "&"
"pants cmd1 ... + cmd2 ..." reads as "(do) cmd1 ... (AND) cmd2 ..."

^
like regex "start" of line or git "parent commit"
"pants cmd1 ... ^ cmd2 ..." reads as "(start) cmd1 ... (start) cmd2 ..."

';' (like xargs)
I don't actually like the xargs interface, but it is interesting prior art.

-and (like find)
This is probably too verbose. Still interesting to consider.

I suppose I could get used to using @ as the separator. I would just read this as "at global" or "at the top" or something like that.

What led you to select @?

It was the first character I thought of that isn't interpreted by the shell... So semicolon is out. But I agree that + would be more evocative, so I'm down with that.

src/rust/pants_ng/options/src/pants_invocation.rs

cognifloyd · 2025-10-27T00:09:03Z

src/rust/pants_ng/options/src/pants_invocation_tests.rs

+    );
+
+    assert_eq!(
+        mk_invocation("pants --foo cmd --bar=baz").unwrap(),


I wonder if we could use a special character at the beginning of a flag to say "this is a global flag".

Maybe , the @ would be a good "global flag" marker. So people (like me) can through a global flag (like keep-sandboxes on the end of a command from shell history like this

pants --foo CMD --bar=baz --@keep-sandboxes=always

Or ^:

pants --foo CMD --bar=baz --^keep-sandboxes=always

Clearly, that could only work before any passthrough args (before --).

Well, we could also say that any flags after specs are global (instead of an error), without any special syntax? There is nothing else they can be.

I assume the idea is to be able to tack global flags on at the end of an existing cmd line?

I assume the idea is to be able to tack global flags on at the end of an existing cmd line?

Yeah. Some debug or log level commands on the end of a failed command can be really helpful for me.

Either going up in shell history or copy/paste from CI and then tack on the debug arg(s) (which is often --keep-sandboxes).

Well, we could also say that any flags after specs are global (instead of an error), without any special syntax? There is nothing else they can be.

Huh. I suppose that would work. Cool. I like it! Would anyone find that confusing or surprising?

benjyw · 2025-10-27T20:10:14Z

I'm bike shedding here. I don't intend to nitpick or be mean or whatever. Just thinking through what it would be like to use this.

Thank you for working on this. And _ng seems like a fine non-committal suffix.

Your comments are valuable and I don't see them as nitpicking, and I certainly don't detect meanness, so don't worry about it.

cognifloyd · 2025-10-28T01:15:53Z

src/rust/pants_ng/options/src/pants_invocation.rs

+        let mut passthru = None;
+        if let Some(s) = unconsumed_args.next() {
+            if s == "--" {


Handling global args after specs would mean adding a consume_flags call right before this passthrough arg handling, right?

Should something like that go in a follow-up (if we do it)?

Eh, it was easy enough to do so I've done it.

cburroughs

Thanks for putting this together.

This parser instead requires specs to contain a filepath separator,
anything else is a flag or a command or a subcommand, and those
are easily distinguished by syntax alone. A spec path in the root dir
can be prefixed with ./ to meet this requirement.

So instead of src::, it would be something like ./src::?

cburroughs · 2025-10-28T02:41:24Z

src/rust/pants_ng/options/src/pants_invocation_tests.rs

+#[test]
+fn test_multiple_commands_and_subcommands() {
+    assert_eq!(
+        mk_invocation("pants cmd1 subcmd1 + cmd2 + cmd3 subcmd3").unwrap(),


So obviously | as a character is out, but I'm wondering if these should be thought of as "do all of this and then do the next one", or "maybe 'line' by 'line' depending on the command".

Thinking of #10542 for example

Oh. Different characters for sequential and concurrent goals? I was thinking only of sequential goals like the current interface. Adding the possibility of concurrency or "goal scripting" opens a whole new works of possibilities. 🤯

I think it would be best if these were always concurrent, no? You can get sequential behavior by running pants multiple times.

cburroughs · 2025-10-28T02:42:07Z

src/rust/pants_ng/options/src/pants_invocation_tests.rs

+    );
+
+    assert_eq!(
+        mk_invocation("pants --global-flag cmd1 --cmd1-flag subcmd1 --subcmd1-flag + cmd2 --cmd2-flag + cmd3 --cmd3-flag subcmd3 --subcmd3-flag path/to/spec -- passthru").unwrap(),


--cmd1-flag should be read as "an example flag used by --cmd1, not a required prefix, correct?

Yes, exactly.

benjyw · 2025-10-30T02:33:16Z

Thanks for putting this together.

This parser instead requires specs to contain a filepath separator,
anything else is a flag or a command or a subcommand, and those
are easily distinguished by syntax alone. A spec path in the root dir
can be prefixed with ./ to meet this requirement.

So instead of src::, it would be something like ./src::?

Exactly

cognifloyd

Great start. Looking forward to where this might go.

cognifloyd · 2025-11-03T03:29:10Z

src/rust/pants_ng/options/src/pants_invocation.rs

+
+        #[inline]
+        fn is_spec(s: &str) -> bool {
+            s.contains(path::MAIN_SEPARATOR)


I don't know rust. When we support Windows at some point, will this be \ on Windows instead of /? Or will it take both? Or does it always use the *nix /?

This will be backslash on Windows, yep: https://doc.rust-lang.org/std/path/constant.MAIN_SEPARATOR.html

Should we support both on Windows?

I think not at first. Maybe if there is demand for it. But we are a thousand other steps away from even evaluating that demand...

benjyw added release-notes:not-required [CI] PR doesn't require mention in release notes pants_ng labels Oct 26, 2025

benjyw requested review from cburroughs and sureshjoshi October 26, 2025 17:56

benjyw added 2 commits October 26, 2025 15:12

Merge branch 'main' into pants_ng_cli

77a7c4e

lint

f5fecba

cognifloyd reviewed Oct 27, 2025

View reviewed changes

benjyw added 2 commits October 27, 2025 13:12

Merge branch 'main' into pants_ng_cli

e1cd4c1

Address code review comments

79ddb98

benjyw force-pushed the pants_ng_cli branch from 95d6420 to 79ddb98 Compare October 27, 2025 20:47

cognifloyd reviewed Oct 28, 2025

View reviewed changes

cburroughs reviewed Oct 28, 2025

View reviewed changes

benjyw requested review from cburroughs and cognifloyd October 30, 2025 02:33

benjyw added 2 commits November 2, 2025 10:12

Merge branch 'main' into pants_ng_cli

a99c33f

Support global flags after specs

ac7b36d

cognifloyd approved these changes Nov 3, 2025

View reviewed changes

benjyw merged commit 89462b7 into pantsbuild:main Nov 3, 2025
54 of 56 checks passed

benjyw deleted the pants_ng_cli branch November 3, 2025 21:49

Uh oh!

A new "pants next-gen" command-line parser. #22808

A new "pants next-gen" command-line parser. #22808

Uh oh!

Conversation

benjyw commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sureshjoshi commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benjyw commented Oct 26, 2025

Uh oh!

cognifloyd left a comment

Choose a reason for hiding this comment

Uh oh!

cognifloyd Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjyw commented Oct 27, 2025

Uh oh!

cognifloyd Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cburroughs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjyw commented Oct 30, 2025

Uh oh!

cognifloyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benjyw commented Oct 26, 2025 •

edited

Loading

sureshjoshi commented Oct 26, 2025 •

edited

Loading

cognifloyd Oct 26, 2025 •

edited

Loading

cognifloyd Oct 28, 2025 •

edited

Loading