Doctests don't work in bin targets, non-public items

[kornelski]

It's surprising that doctests sometimes don't run at all, and there's no warning about this.

cargo new foo --bin

/// ```rust
/// assert!(false);
/// ```
fn main() {
    println!("Hello, world!");
}

cargo test --all

I'd expect the above to fail, but it looks like the doc-comment code is never ran. Even if I explicitly enable doctests for the binary:

[[bin]]
name = "foo"
path = "src/main.rs"
doctest = true

they still aren't run.

(moved from rust-lang/cargo#5477)

[GuillaumeGomez]

It sounds weird to have documentation examples in a bin target, don't you think? There is no documentation to generate so why would there be documentation's code to run?

[kornelski]

This bug is based on feedback from a user.

Cargo makes new users create bin targets by default, so the first time a new Rust user tries doc tests, it's probably in a bin.

There's a mismatch between what rustdoc does (tests publicly-visible documentation) and expectations (that tests in doccomments, in general, are run).

[GuillaumeGomez]

Maybe we give bad information ahead? I wonder if users know the difference between comments and doc comments... For now I don't see this as a rustdoc bug but as a misunderstanding that should be resolved in tutorials/books.

cc @rust-lang/docs

[vi]

Maybe doctests should just run unconditionally?

The code for a non-public bin-crate item may end up in a public, lib-crate item later, so what's wrong having documentation and tests a bit earlier?

[QuietMisdreavus]

The problem with running doctests on bin targets lies with how doctests work. The way rustdoc creates doctests is by building the code sample as a bin target of its own, and linking it against your crate. If "your crate" is really an executable, there's nothing to link against.

If you run cargo test --doc --verbose, you can see that Cargo needs to build your crate ahead of time, and pass the rlib to rustdoc:

[misdreavus@tonberry asdf]$ cargo test --doc --verbose
   Compiling asdf v0.1.0 (file:///home/misdreavus/git/asdf)
     Running `rustc --crate-name asdf src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=31e1a62d36de9a99 -C extra-filename=-31e1a62d36de9a99 --out-dir /home/misdreavus/git/asdf/target/debug/deps -C incremental=/home/misdreavus/git/asdf/target/debug/incremental -L dependency=/home/misdreavus/git/asdf/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.79 secs
   Doc-tests asdf
     Running `rustdoc --test /home/misdreavus/git/asdf/src/lib.rs --crate-name asdf -L dependency=/home/misdreavus/git/asdf/target/debug/deps -L dependency=/home/misdreavus/git/asdf/target/debug/deps --extern asdf=/home/misdreavus/git/asdf/target/debug/deps/libasdf-31e1a62d36de9a99.rlib`

running 1 test
test src/lib.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Semantically, i find nothing wrong with allowing doctests in bin crates. However, to pull this off, we would need to tell Cargo to compile bin crates as libs, then hand that rlib to rustdoc for testing. There's nothing stopping us from doing that right now:

[misdreavus@tonberry asdf]$ rustc --crate-type lib --crate-name asdf src/main.rs
[misdreavus@tonberry asdf]$ rustdoc --test --crate-name asdf src/main.rs --extern asdf=libasdf.rlib

running 1 test
test src/main.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cc @rust-lang/cargo

[kornelski]

Wow, that's a nice surprise that it works. I was afraid that duplicate main would be a problem, but I guess rlib namespaces it!

[QuietMisdreavus]

Yeah, the main in your bin goes inside the lib crate, so it's just a regular function at that point. I did have to tag it with #[allow(dead_code)] to silence a warning, but otherwise it's all good.

[PhilipDaniels]

I just hit this today. I came here after finding rust-lang/cargo#2009 in a Google search, which mentions that it is 'addressed in the documentation'. I suspect it may have been at one point but it doesn't appear to be now. I checked

https://doc.rust-lang.org/book/ch11-01-writing-tests.html

and from there the link

https://doc.rust-lang.org/book/ch14-02-publishing-to-crates-io.html#documentation-comments-as-tests

(Of course it's possible its mentioned somewhere and I missed it. A very careful reading of the surrounding text implies it, because it's all talking about libs, but nowhere I see is it mentioned explicitly.)

[joshtriplett]

I ran into this today, and found it confusing. I would love to be able to write doctests in the modules of my binary crate.

What would it take to make that work out-of-the-box?

[ehuss]

I can think of a few hurdles that someone would need to address:

• How to resolve symbols in a binary? Doctests behave as-if they are a separate crate linking in the library. But a binary is normally not used a library, so it has no name to refer to. For example:

/// ```
/// do_something();
/// ```
fn do_something() {}

fn main() {}

Here, do_something(); would fail because it is not in scope. You could try to scope it with the name of the binary (mything::do_something();), but then you have two new problems: what if you have a lib target of the same name? And, deal with the fact that you normally don't export any public items in a binary.

• How to deal with lack of public exports? Building a binary as a library will result in a lot of unused warnings. And since rustdoc imports the crate, usually nothing will be publicly exported, so you won't be able to access anything.

One (rough) idea to address this would be to fundamentally change how rustdoc --test works. Instead of creating a synthetic test that links to a library, embed the tests directly in the module. So it would rewrite the above example to:

fn do_something() {}

#[test]
fn do_something_doctest() {
    do_something();
}

So all local (private) symbols would be in scope.

That's a pretty radical difference, so I'm not sure if it would fly (would probably be confusing if there are two styles of doctests).

Also, I think it's worth examining if this is a good idea from the start. Documenting binaries still seems like a strange concept to me. Why can't you just move all the code to a library?