This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1429

test-cstore: run native Python tests directly
ClosedPublic

Authored by quark on Nov 15 2017, 4:38 PM.
Tags
None
Subscribers
durham

Details

Reviewers
durham
Group Reviewers
Restricted Project
Commits
rFBHGXcfe92441f1c4: test-cstore: run native Python tests directly
Summary

Previously the test sets up LD_LIBRARY_PATH and PYTHONPATH, then runs
Python tests.

Within Python code, setting sys.path would achieve the same effect of
setting PYTHONPATH. For LD_LIBRARY_PATH, it's necessary for C libraries.
But the only C library that cstore depends on is lz4, which is supposed to
use the system version. There is no C library provided by this repo -
features like sha1 are compiled in cstore.so.

Therefore it's unnecessary to have a separate .t file wrapping .py
tests. Let's just use .py tests directly.

Test Plan

./script/unit.py

Make a temporary change to cdatapack.c so it fails unconditionally in
open_datapack. Build the repo in different ways: make local and
python2 setup.py build_clib build_ext. Then run the test by using
$HG_CREW/tests/run-tests.py -l test-remotefilelog-datapack.py and make
sure it fails with the expected exception.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

quark created this revision.Nov 15 2017, 4:38 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 15 2017, 4:38 PM
durham accepted this revision.Nov 16 2017, 1:43 PM
durham added a subscriber: durham.

Could you test that the tests are still executing the built c datapack stuff and not the system one? Just add an exception to some C code and run the tests and make sure they fail.

This revision is now accepted and ready to land.Nov 16 2017, 1:43 PM
durham added a comment.Nov 16 2017, 1:44 PM

In particular, make sure it works outside of hg-dev

quark edited the test plan for this revision. (Show Details)Nov 16 2017, 1:55 PM
quark updated this revision to Diff 3571.
Closed by commit rFBHGXcfe92441f1c4: test-cstore: run native Python tests directly (authored by quark). · Explain WhyNov 20 2017, 9:54 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 3534

tests/cstore-datapackstore.py

  • This file was moved to tests/test-cstore-datapackstore.py.

tests/cstore-treemanifest.py

  • This file was moved to tests/test-cstore-treemanifest.py.

tests/cstore-uniondatapackstore.py

  • This file was moved to tests/test-cstore-uniondatapackstore.py.

tests/pythonpath.py

  • This file was added.
from __future__ import absolute_import
import glob
import os
import sys
reporoot = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def globrelative(path):
return glob.glob(os.path.join(reporoot, path))
def setcstorepath():
sys.path[0:0] = (
# make local
[reporoot] +
# python2 setup.py build_ext
globrelative('build/lib*') +
# rpmbuild
globrelative('../rpmbuild/BUILD/fb-mercurial-ext-*/build/lib.*')
)

tests/remotefilelog-datapack.py

  • This file was moved to tests/test-remotefilelog-datapack.py.

tests/remotefilelog-histpack.py

  • This file was moved to tests/test-remotefilelog-histpack.py.

tests/test-check-py3-compat-hg.t

remotefilelog/wirepack.py not using absolute_import remotefilelog/wirepack.py not using absolute_import
scripts/lint.py not using absolute_import scripts/lint.py not using absolute_import
scripts/lint.py requires print_function scripts/lint.py requires print_function
scripts/unit.py not using absolute_import scripts/unit.py not using absolute_import
scripts/utils.py not using absolute_import scripts/utils.py not using absolute_import
setup.py not using absolute_import setup.py not using absolute_import
tests/bundlerepologger.py not using absolute_import tests/bundlerepologger.py not using absolute_import
tests/conduithttp.py not using absolute_import tests/conduithttp.py not using absolute_import
tests/cstore-datapackstore.py not using absolute_import
tests/cstore-treemanifest.py not using absolute_import
tests/cstore-uniondatapackstore.py not using absolute_import
tests/dummyext1.py not using absolute_import tests/dummyext1.py not using absolute_import
tests/dummyext2.py not using absolute_import tests/dummyext2.py not using absolute_import
tests/get-with-headers.py not using absolute_import tests/get-with-headers.py not using absolute_import
tests/get-with-headers.py requires print_function tests/get-with-headers.py requires print_function
tests/getflogheads.py not using absolute_import tests/getflogheads.py not using absolute_import
tests/heredoctest.py not using absolute_import tests/heredoctest.py not using absolute_import
tests/heredoctest.py requires print_function tests/heredoctest.py requires print_function
tests/killdaemons.py not using absolute_import tests/killdaemons.py not using absolute_import
tests/ls-l.py not using absolute_import tests/ls-l.py not using absolute_import
tests/ls-l.py requires print_function tests/ls-l.py requires print_function
tests/perftest.py not using absolute_import tests/perftest.py not using absolute_import
tests/perftest.py requires print_function tests/perftest.py requires print_function
tests/remotefilelog-datapack.py not using absolute_import
tests/remotefilelog-datapack.py requires print_function
tests/remotefilelog-histpack.py not using absolute_import
tests/test-absorb-filefixupstate.py not using absolute_import tests/test-absorb-filefixupstate.py not using absolute_import
tests/test-extutil.py not using absolute_import tests/test-extutil.py not using absolute_import
tests/test-fastmanifest.py not using absolute_import tests/test-fastmanifest.py not using absolute_import
tests/test-generic-bisect.py not using absolute_import tests/test-generic-bisect.py not using absolute_import
tests/test-patchpython.py not using absolute_import tests/test-patchpython.py not using absolute_import
tests/test-sshaskpass.py not using absolute_import tests/test-sshaskpass.py not using absolute_import
tests/treemanifest_correctness.py not using absolute_import tests/treemanifest_correctness.py not using absolute_import
tests/waitforfile.py not using absolute_import tests/waitforfile.py not using absolute_import
treemanifest/__init__.py not using absolute_import treemanifest/__init__.py not using absolute_import

tests/test-cstore-datapackstore.py

  • This file was moved from tests/cstore-datapackstore.py.
#!/usr/bin/env python2.7 #!/usr/bin/env python2.7
from __future__ import absolute_import
import hashlib import hashlib
import os import os
import random import random
import shutil import shutil
import sys import sys
import tempfile import tempfile
import time import time
import unittest import unittest
import silenttestrunner import silenttestrunner
# Load the local cstore, not the system one # Load the local cstore, not the system one
fullpath = os.path.join(os.getcwd(), __file__) sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')]
sys.path.insert(0, os.path.dirname(os.path.dirname(fullpath))) import pythonpath
pythonpath.setcstorepath()
from cstore import ( from cstore import (
datapackstore, datapackstore,
) )
from remotefilelog.datapack import ( from remotefilelog.datapack import (
fastdatapack, fastdatapack,
mutabledatapack, mutabledatapack,

tests/test-cstore-treemanifest.py

  • This file was moved from tests/cstore-treemanifest.py.
#!/usr/bin/env python2.7 #!/usr/bin/env python2.7
from __future__ import absolute_import
import os import os
import random import random
import sys import sys
import unittest import unittest
import silenttestrunner import silenttestrunner
# Add the repo root to the path so we can find the built ctreemanifest # Add the repo root to the path so we can find the built ctreemanifest
fullpath = os.path.join(os.getcwd(), __file__) sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')]
sys.path.insert(0, os.path.dirname(os.path.dirname(fullpath))) import pythonpath
pythonpath.setcstorepath()
import cstore import cstore
from mercurial import ( from mercurial import (
manifest, manifest,
match as matchmod match as matchmod
) )
from mercurial.node import hex, nullid from mercurial.node import hex, nullid

tests/test-cstore-uniondatapackstore.py

  • This file was moved from tests/cstore-uniondatapackstore.py.
#!/usr/bin/env python2.7 #!/usr/bin/env python2.7
from __future__ import absolute_import
import hashlib import hashlib
import os import os
import random import random
import shutil import shutil
import sys import sys
import tempfile import tempfile
import unittest import unittest
import silenttestrunner import silenttestrunner
# Load the local cstore, not the system one # Load the local cstore, not the system one
fullpath = os.path.join(os.getcwd(), __file__) sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')]
sys.path.insert(0, os.path.dirname(os.path.dirname(fullpath))) import pythonpath
pythonpath.setcstorepath()
from cstore import ( from cstore import (
datapackstore, datapackstore,
uniondatapackstore, uniondatapackstore,
) )
from remotefilelog.datapack import ( from remotefilelog.datapack import (
datapack, datapack,

tests/test-cstore.t

  • This file was deleted.
Get build path for native extensions
- Do this as a separate script so we can eat the ls errors
$ cat >> $TESTTMP/getbuildpath.sh <<EOF
> BUILD_PATH="`ls -d $TESTDIR/`"
> BUILD_PATH="\$BUILD_PATH:`ls -d $TESTDIR/../build/lib* 2> /dev/null`"
> BUILD_PATH="\$BUILD_PATH:`ls -d $TESTDIR/../../rpmbuild/BUILD/fb-mercurial-ext-*/build/lib.*/ 2> /dev/null`"
> echo "\$BUILD_PATH"
> EOF
$ chmod a+x $TESTTMP/getbuildpath.sh
$ BUILD_PATH="`$TESTTMP/getbuildpath.sh`"
$ LD_LIBRARY_PATH="$BUILD_PATH"
$ export LD_LIBRARY_PATH
$ PYTHONPATH="$BUILD_PATH:$PYTHONPATH"
$ export PYTHONPATH
$ $PYTHON $TESTDIR/remotefilelog-datapack.py
$ $PYTHON $TESTDIR/remotefilelog-histpack.py
$ $PYTHON $TESTDIR/cstore-datapackstore.py
$ $PYTHON $TESTDIR/cstore-treemanifest.py
$ $PYTHON $TESTDIR/cstore-uniondatapackstore.py

tests/test-remotefilelog-datapack.py

  • This file was moved from tests/remotefilelog-datapack.py.
#!/usr/bin/env python #!/usr/bin/env python
from __future__ import absolute_import, print_function
import hashlib import hashlib
import os import os
import random import random
import resource import resource
import shutil import shutil
import stat import stat
import struct import struct
import sys import sys
import tempfile import tempfile
import time import time
import unittest import unittest
import silenttestrunner import silenttestrunner
# Load the local remotefilelog, not the system one # Load the local remotefilelog, not the system one
sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')] sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')]
import pythonpath
pythonpath.setcstorepath()
from remotefilelog.datapack import ( from remotefilelog.datapack import (
datapack, datapack,
datapackstore, datapackstore,
fastdatapack, fastdatapack,
mutabledatapack, mutabledatapack,
) )
from remotefilelog.basepack import ( from remotefilelog.basepack import (
% (resource.getrlimit(resource.RLIMIT_NOFILE),)) % (resource.getrlimit(resource.RLIMIT_NOFILE),))
# Print debug information about what files are opened by the # Print debug information about what files are opened by the
# current process. # current process.
os.system('lsof -p %s' % os.getpid()) os.system('lsof -p %s' % os.getpid())
# perf test off by default since it's slow # perf test off by default since it's slow
def _testIndexPerf(self): def _testIndexPerf(self):
random.seed(0) random.seed(0)
print "Multi-get perf test" print("Multi-get perf test")
packsizes = [ packsizes = [
100, 100,
10000, 10000,
100000, 100000,
500000, 500000,
1000000, 1000000,
3000000, 3000000,
] ]
start = time.time() start = time.time()
pack.getmissing(findnodes[:lookupsize]) pack.getmissing(findnodes[:lookupsize])
elapsed = time.time() - start elapsed = time.time() - start
print ("%s pack %s lookups = %0.04f" % print ("%s pack %s lookups = %0.04f" %
(('%s' % packsize).rjust(7), (('%s' % packsize).rjust(7),
('%s' % lookupsize).rjust(7), ('%s' % lookupsize).rjust(7),
elapsed)) elapsed))
print "" print("")
gc.enable() gc.enable()
# The perf test is meant to produce output, so we always fail the test # The perf test is meant to produce output, so we always fail the test
# so the user sees the output. # so the user sees the output.
raise RuntimeError("perf test always fails") raise RuntimeError("perf test always fails")
class datapacktests(datapacktestsbase, unittest.TestCase): class datapacktests(datapacktestsbase, unittest.TestCase):
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):

tests/test-remotefilelog-histpack.py

  • This file was moved from tests/remotefilelog-histpack.py.
#!/usr/bin/env python #!/usr/bin/env python
from __future__ import absolute_import
import hashlib import hashlib
import os import os
import random import random
import shutil import shutil
import stat import stat
import struct import struct
import sys import sys
import tempfile import tempfile
import unittest import unittest
import silenttestrunner import silenttestrunner
# Load the local remotefilelog, not the system one # Load the local remotefilelog, not the system one
sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')] sys.path[0:0] = [os.path.join(os.path.dirname(__file__), '..')]
import pythonpath
pythonpath.setcstorepath()
from remotefilelog.historypack import historypack, mutablehistorypack from remotefilelog.historypack import historypack, mutablehistorypack
from mercurial.node import nullid from mercurial.node import nullid
import mercurial.ui import mercurial.ui
from remotefilelog.basepack import ( from remotefilelog.basepack import (
SMALLFANOUTCUTOFF, SMALLFANOUTCUTOFF,
LARGEFANOUTPREFIX, LARGEFANOUTPREFIX,
tests/test-cstore-uniondatapackstore.py