error when ABI does not support guaranteed tail calls

[folkertdev]

Some ABIs cannot support guaranteed tail calls. There isn't really an exhaustive list, so this is a best effort. Conveniently, we already disallow calling most of these directly anyway. The only exception that I was able to trigger an LLVM assertion with so far was cmse-nonsecure-entry.

For that calling convention, LLVM specifically notes that (guaranteed) tail calls cannot be supported:

https://github.com/llvm/llvm-project/blob/28dbbba6c3a4e026e085c48cc022cb97b5d8bc6d/llvm/lib/Target/ARM/ARMISelLowering.cpp#L2331-L2335

---

I have some doubts about the implementation here though. I think it would be nicer to use CanonAbi, and move the become ABI check into rustc_hir_typeck, similar to check_call_abi:

rust/compiler/rustc_hir_typeck/src/callee.rs

Lines 157 to 194 in d6deffe

	pub(crate) fn check_call_abi(&self, abi: ExternAbi, span: Span) {
	let canon_abi = match AbiMap::from_target(&self.sess().target).canonize_abi(abi, false) {
	AbiMapping::Direct(canon_abi) | AbiMapping::Deprecated(canon_abi) => canon_abi,
	AbiMapping::Invalid => {
	// This should be reported elsewhere, but we want to taint this body
	// so that we don't try to evaluate calls to ABIs that are invalid.
	let guar = self.dcx().span_delayed_bug(
	span,
	format!("invalid abi for platform should have reported an error: {abi}"),
	);
	self.set_tainted_by_errors(guar);
	return;
	}
	};
	let valid = match canon_abi {
	// Rust doesn't know how to call functions with this ABI.
	CanonAbi::Custom => false,
	// These is an entry point for the host, and cannot be called on the GPU.
	CanonAbi::GpuKernel => false,
	// The interrupt ABIs should only be called by the CPU. They have complex
	// pre- and postconditions, and can use non-standard instructions like `iret` on x86.
	CanonAbi::Interrupt(_) => false,
	CanonAbi::C
	| CanonAbi::Rust
	| CanonAbi::RustCold
	| CanonAbi::Arm(_)
	| CanonAbi::X86(_) => true,
	};
	if !valid {
	let err = crate::errors::AbiCannotBeCalled { span, abi };
	self.tcx.dcx().emit_err(err);
	}
	}

Both the check for whether an ABI is callable and whether it supports guaranteed tail calls can then be methods (containing exhaustive matches) on CanonAbi. I'm however not sure

• if the ABI checks are deliberately only performed when constructing MIR
• what assumptions can be made about the call expression in check_expr_become, it looks like currently the check that the "argument" to become is a function call also only occurs later during MIR construction

Are there issues with validating the ABI earlier in rustc_hir_typeck that I'm overlooking? I believe that we should already know the call's ABI and whether it is c-variadic at that point.

cc @workingjubilee for CanonAbi, @davidtwco for cmse
r? @WaffleLapkin

[rustbot]

WaffleLapkin is not on the review rotation at the moment.
They may take a while to respond.