Is there any reason this can't be done using Rc<RefCell<DirsMultiset>>? I have an example here: https://rust.godbolt.org/z/MNNR_F

It uses Rc and RefCell however it does need to do some unsafe work to keep the RefCell borrowed for longer than would otherwise be easy to do. The advantages here are

1. Lifetime is automatically managed by the Rc. This ensures that our parent data doesn't get deleted while the iterator is still alive.
2. The linked implementation uses a mutable borrow. However it would be easy to make an immutable version which would allow multiple iterators over the same "container".
3. Mutable access is managed by the RefCell so we don't need to do our own.

In D6631#97216, @kevincox wrote:

Is there any reason this can't be done using Rc<RefCell<DirsMultiset>>? I have an example here: https://rust.godbolt.org/z/MNNR_F

I see now that since python is managing the reference it does make some sense to do it this way.

@kevincox I've renamed the file since it no longer just contains macros.

I've moved whatever I could move to separate structs and used Cell instead of RefCell on those scalar types. I'm not so keen on using UnsafeCell, espacially since the target py_class!-resulting struct could be using RefCell-specific APIs.

I'm not too sure on the terminology, so tell me if you have better ideas than mine.

Later down the line, I could see a trait to help define the interface for shared structs.

it still feels like we are kinda implementing a second RefCell anyways

We all share that feeling, I think. Let's hope we'll be able to circumvent that in some future.

but I think this code is quite clean.

Thanks, @Alphare is currently on leave. I'm sure he'll appreciate when he comes back.

+impl PySharedState {
+ pub fn borrow_mut<'a, T>(
+ &'a self,
+ py: Python<'a>,
+ pyrefmut: RefMut<'a, T>,
+ ) -> PyResult<PyRefMut<'a, T>> {
+ if self.mutably_borrowed.get() {
+ return Err(AlreadyBorrowed::new(
+ py,
+ "Cannot borrow mutably while there exists another \
+ mutable reference in a Python object",
+ ));
+ }
+ match self.leak_count.get() {
+ 0 => {
+ self.mutably_borrowed.replace(true);
+ Ok(PyRefMut::new(py, pyrefmut, self))
+ }
+ TODO
+ For now, this works differently than Python references
+ in the case of iterators.
+ Python does not complain when the data an iterator
+ points to is modified if the iterator is never used
+ afterwards.
+ Here, we are stricter than this by refusing to give a
+ mutable reference if it is already borrowed.
+ While the additional safety might be argued for, it
+ breaks valid programming patterns in Python and we need
+ to fix this issue down the line.
+ _ => Err(AlreadyBorrowed::new(
+ py,
+ "Cannot borrow mutably while there are \
+ immutable references in Python objects",
+ )),
+ }
+ }
+
+ / Return a reference to the wrapped data with an artificial static
+ / lifetime.
+ / We need to be protected by the GIL for thread-safety.
+ pub fn leak_immutable<T>(
+ &self,
+ py: Python,
+ data: &RefCell<T>,
+ ) -> PyResult<&'static T> {
+ if self.mutably_borrowed.get() {
+ return Err(AlreadyBorrowed::new(
+ py,
+ "Cannot borrow immutably while there is a \
+ mutable reference in Python objects",
+ ));
+ }
+ let ptr = data.as_ptr();
+ self.leak_count.replace(self.leak_count.get() + 1);
+ unsafe { Ok(&*ptr) }
+ }

This leak_immutable() looks potentially unsafe since it can extend the
lifetime of an arbitrary object, data.

+macro_rules! py_shared_ref {
+ (
+ $name: ident,
+ $inner_struct: ident,
+ $data_member: ident,
+ $leaked: ident,
+ ) => {
+ impl $name {
+ fn borrow_mut<'a>(
+ &'a self,
+ py: Python<'a>,
+ ) -> PyResult<crate::ref_sharing::PyRefMut<'a, $inner_struct>>
+ {
+ self.py_shared_state(py)
+ .borrow_mut(py, self.$data_member(py).borrow_mut())
+ }
+
+ fn leak_immutable<'a>(
+ &'a self,
+ py: Python<'a>,
+ ) -> PyResult<&'static $inner_struct> {
+ self.py_shared_state(py)
+ .leak_immutable(py, self.$data_member(py))
+ }
+ }

Is it okay to directly borrow/borrow_mut() the inner object?

For example, we can still do self.inner(py).borrow_mut() (in place of
self.borrow_mut(py)) while the inner object is leaked, which I think is
unsafe.

If I understand the py-shared logic correctly, PySharedState needs to own
the inner object to guarantee that any public operations are safe.