Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Formatchange #1

Open
wants to merge 80 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
50d686f
Added conversion function to change dict format of graphs to use ints…
Jul 31, 2015
dc68da7
Finished converting to module and cleaning up imports, started adding…
Aug 3, 2015
65614ba
Added files from directory restructure
Aug 3, 2015
8a0672a
Moved conversion functions into their own module. Refactored and rem…
Aug 4, 2015
47fc895
Finished cursorary cleanup of unused and duplicate functions. All co…
Aug 5, 2015
b92bb3c
Undersampling and tests converted
Aug 6, 2015
6e9712c
converted bfutils, comparison, conversions to new format
Aug 6, 2015
e9adf2a
Fixed pickle loading to happen lazily so imports dont take forever
Aug 7, 2015
c16b36e
More files converted
Aug 8, 2015
3c09028
Working on traversal.py
Aug 8, 2015
d2a9609
Fixed import
Aug 13, 2015
734d5ce
Added tests and removed more unused code
Aug 13, 2015
86072b5
Removed more unused code
Aug 13, 2015
a514a76
Removed more unused code
Aug 13, 2015
b28ba60
Added a new test
Aug 13, 2015
01e821f
Added a new tests
Aug 13, 2015
43aea96
Added a new tests
Aug 13, 2015
f46c5c8
Added a new tests
Aug 13, 2015
67e4666
Removed more unused code
Aug 13, 2015
5204c7d
Added a new tests and converted more code
Aug 13, 2015
eba5092
Added a new tests and converted more code
Aug 13, 2015
75f006a
Finished converting tools directory, cleaned up more unused code
Aug 14, 2015
03f9931
Added all dependant zkl files and commented out from gitignore
Aug 14, 2015
8c211cc
Incorporated Sergeys bug fix
Aug 15, 2015
eebd701
Sergeys workaround
Aug 18, 2015
ee86315
Sergeys workaround
Aug 18, 2015
f7a4d9e
Sergeys patch
Aug 18, 2015
f04b27c
Fixed test output file after bug was patched by Sergey
Aug 18, 2015
be16ac7
Added back traversal test from branch
Aug 18, 2015
938ca13
Traversal test passing with old format
Aug 18, 2015
545cdf8
finished converting all scripts
Aug 18, 2015
de25c0d
Reverted workaround
Aug 18, 2015
8b79a3b
Reverted workaround
Aug 18, 2015
079cef5
Somehow the bug fix merge didnt take, so added it back in
Aug 18, 2015
65521da
added import of permutations
sergeyplis Aug 19, 2015
c094ee2
option 1 for the SAT4U project is implemented and now
sergeyplis Sep 11, 2015
f640744
option 2 is also implemented
sergeyplis Sep 11, 2015
78d2abc
changes to account for clingo and conversion to clingo file
sergeyplis Sep 28, 2015
2a8b12b
scripting changes
sergeyplis Sep 28, 2015
c31202a
added an explicit threshold parameter
sergeyplis Oct 7, 2015
dd06840
using multiple linear regression for the test
sergeyplis Oct 7, 2015
2d8cd5e
just the working process of parameter playing
sergeyplis Oct 7, 2015
c7dd31e
weighted version of clingo
sergeyplis Oct 12, 2015
c45079b
scropting and linear models
sergeyplis Oct 24, 2015
99bd918
scripts
sergeyplis Oct 24, 2015
d0976be
getting rid of the gmpy dependency
sergeyplis Oct 27, 2015
9ed835c
another minus gmpy dependency
sergeyplis Oct 27, 2015
e8ccfbe
a slight tweak to conversion g2num
sergeyplis Oct 27, 2015
60b82b7
a, hopefully, faster g2num
sergeyplis Feb 24, 2017
0ca09ad
optimizations
sergeyplis Feb 24, 2017
de64406
getting rid of gmpy
sergeyplis Feb 24, 2017
017437f
just using the script and playing with parameters
sergeyplis Feb 24, 2017
9c7b049
importing sys module and corrected the path to the figure template
sergeyplis Feb 24, 2017
229e158
got rid of gmpy dependency
sergeyplis Feb 24, 2017
3a33de0
and even the commented out version
sergeyplis Feb 24, 2017
0c6098e
distribution to sample from
sergeyplis Feb 24, 2017
80e4808
plotting functions for rasl paper
sergeyplis Feb 24, 2017
15832aa
clingo
sergeyplis Feb 24, 2017
59ec611
Merge blackbolt:/na/homes/splis/soft/src/dev/craft/ian/gunfolds into …
sergeyplis Feb 25, 2017
21cde8e
removed the old representation function
sergeyplis Feb 25, 2017
bfa20f1
clingo solver for MSL algorithm
sergeyplis Feb 25, 2017
2e4f317
removed the weighted version hooks
sergeyplis Feb 25, 2017
e6c87f9
help string to dens2edgenum
sergeyplis Feb 25, 2017
c75bb60
added python-igraph and removed igraph from requirements
sergeyplis Feb 26, 2017
3d0fe4b
a function to convert the old to new representation
sergeyplis Feb 27, 2017
f673493
made nameid work
sergeyplis Feb 27, 2017
5e1386e
fixed incorrect linebreak
sergeyplis Feb 27, 2017
791c4ad
adding close_fds=True and also closing the stdout manually
sergeyplis Mar 1, 2017
031b638
was closing too early
sergeyplis Mar 1, 2017
4b7e33b
fixed a bug: a graph was incorrectly converted to a vector :(
sergeyplis Mar 1, 2017
bf98382
Merge branch 'formatchange'
sergeyplis Mar 3, 2017
63172b2
removing old2new, as it is already define in conversions
sergeyplis Mar 4, 2017
b34d6ec
convert Ian's representation to the old one
sergeyplis Mar 4, 2017
63753f2
turned out one does not need prefix '0b' for int conversion
sergeyplis Mar 6, 2017
8864026
fixed a bug in increment() function
sergeyplis Mar 6, 2017
ff11e6c
replaced conversion routines g2num and num2CG, the old ones did not m…
sergeyplis Mar 6, 2017
f506ffb
graph_tool binding to plot compressed graphs
sergeyplis Mar 6, 2017
35c93a8
Merged upstream changes from sergey's github
Mar 6, 2017
eb4d8ad
Merged upstream changes from sergey's github
Mar 6, 2017
a1abd31
Merged Sergey's gitlab changes
Mar 6, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merged upstream changes from sergey's github
Ian Beaver committed Mar 6, 2017

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 35c93a8a0b0da905e99d6bd19b40da1f43bfeb50
6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 10 additions & 6 deletions gunfolds/scripts/bf_Gu_size.py
Original file line number Diff line number Diff line change
@@ -52,20 +52,24 @@ def wrapper_non_eliminate(fold):
print template.format(*(' u', ' all_uniq', ' unique', ' seen_Gu', ' converge', ' uconverge'))
cumset = set()
clen = 0
for s in xrange(23):
for s in xrange(100):

results = pool.map(functools.partial(wrapper_u, steps=s),
range(2 ** (nodes ** 2)))
results=pool.map(functools.partial(wrapper_u, steps=s),
range(2**(nodes**2)))

converged = len([e for e in results if e == []])
converged = len([e for e in results if e[1]==[]])
notconverged = len(results) - converged

results = filter(None, results)
if notconverged < 100 and notconverged > 0:
survivors = [e for e in results if e[1]]
if notconverged == 0: break
results = filter(lambda (x): x[1] != [], results)

r = set(results)
d = r.difference(cumset)
cumset = cumset.union(r)

cl = 2 ** (nodes ** 2) - len(results) - clen
cl = 2**(nodes**2) - len(results) - clen
clen += cl

print template.format(*(s, len(r), len(d), len(cumset),
3 changes: 2 additions & 1 deletion gunfolds/scripts/dbnplot.py
Original file line number Diff line number Diff line change
@@ -12,6 +12,7 @@

def listplot(fname, mname='JJ', stl='', width=5):
l = zkl.load(fname)
l = l[:17*10]
y = min(width, len(l))
x = np.int(np.ceil(len(l) / float(y)))
d2l.matrix_list(l, y, x, R=2, w_gap=1, h_gap=2, mname=mname, stl=stl)
@@ -55,7 +56,7 @@ def listplot(fname, mname='JJ', stl='', width=5):
}

# d2l.matrix_unfold(l[0],2,1,R=5, w_gap=1, h_gap=2, mname='TT1')
listplot('list.zkl', width=5)
listplot('list.zkl', width=17)

sys.stdout = sys.__stdout__ # remember to reset sys.stdout!
foo.flush()
2 changes: 1 addition & 1 deletion gunfolds/scripts/g2g1runner.py
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@
import socket
import time

UMAX = 6
UMAX = 1
INPNUM = 1 # number of randomized starts per graph
CAPSIZE = 1000 # stop traversing after growing equivalence class tothis size
REPEATS = 100
15 changes: 8 additions & 7 deletions gunfolds/scripts/rate_agnostic_runner.py
Original file line number Diff line number Diff line change
@@ -10,8 +10,8 @@
import socket
import scipy

KEY = 'ral_async'
UMAX = 4
KEY = 'rasl_il_u2'
UMAX = 2
INPNUM = 1 # number of randomized starts per graph
CAPSIZE = 1000 # stop traversing after growing equivalence class tothis size
REPEATS = 100
@@ -66,9 +66,10 @@ def ra_wrapper(fold, n=10, k=10):
gs = bfutils.call_undersamples(g)
for u in range(1, min([len(gs), UMAX])):
g2 = bfutils.undersample(g, u)
print fold, ': ', traversal.density(g), ':',
print fold,': ',traversal.density(g),':', traversal.density(g2),':',
startTime = int(round(time.time() * 1000))
s = ur.liteqclass(g2, verbose=False, capsize=CAPSIZE)
#s = ur.liteqclass(g2, verbose=False, capsize=CAPSIZE)
s = ur.eqclass(g2)
endTime = int(round(time.time() * 1000))
print len(s), u
l[u] = {'eq': s, 'ms': endTime - startTime}
@@ -133,8 +134,8 @@ def inside_wrapper():
scipy.random.seed()
try:
startTime = int(round(time.time() * 1000))
s = traversal.v2g22g1(g2, capsize=CAPSIZE)
# s = traversal.backtrack_more2(g2, rate=2, capsize=CAPSIZE)
#s = traversal.v2g22g1(g2, capsize=CAPSIZE)
s = traversal.backtrack_more2(g2, rate=2, capsize=CAPSIZE)
endTime = int(round(time.time() * 1000))
print "{:2}: {:8} : {:4} {:10} seconds".\
format(fold, round(gdens, 3), len(s),
@@ -183,7 +184,7 @@ def inside_wrapper():
z = {}
# pool=Pool(processes=PNUM)
for dens in densities[nodes]:
print "{:2}: {:8} : {:10} {:10}".format('id', 'density', 'eq class', 'time')
print "{:2}: {:8} : {:10} : {:10} {:10}".format('id', 'densityi(G)', 'density(H)', 'eq class', 'time')
e = bfutils.dens2edgenum(dens, n=nodes)

multiprocess([[i, nodes, e] for i in range(REPEATS)], PNUM)
46 changes: 23 additions & 23 deletions gunfolds/scripts/svar.py
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@
from gunfolds.tools import linear_model as lm
from gunfolds.tools import pc
from gunfolds.tools import traversal as trv
from gunfolds.tools import unknownrate as ur
from gunfolds.tools import zickle as zkl
import itertools
from multiprocessing import Pool
@@ -22,15 +23,16 @@

NOISE_STD = '0.1'
DEPTH = 2
URATE=3
DIST = 'beta'
BURNIN = 100
SAMPLESIZE = 2000
PARALLEL = True
POSTFIX = '_H'
EST = 'pc'
POSTFIX='_rasl_u'+str(URATE)
EST = 'svar'
INPNUM = 1 # number of randomized starts per graph
CAPSIZE = 100 # stop traversing after growing equivalence class tothis size
REPEATS = 100
REPEATS = 20
PNUM = get_process_count(INPNUM)


@@ -67,23 +69,20 @@ def hamming_neighbors(v, step):


def find_nearest_reachable(g2, max_depth=4):
s = trv.v2g22g1(g2, capsize=CAPSIZE, verbose=False)
# s = trv.edge_backtrack2g1_directed(g2, capsize=CAPSIZE)
s = ur.liteqclass(g2, capsize=CAPSIZE, verbose=False)
if s:
return s
step = 1
n = len(g2)
v = bfu.g2vec(g2)
while True:
l = hamming_neighbors(v, step)
pbar = ProgressBar(
widgets=['neighbors checked @ step ' + str(step) + ': ', Percentage(), ' '], maxval=len(l)).start()
l = hamming_neighbors(v,step)
pbar = ProgressBar(widgets=['neighbors checked @ step '+str(step)+': ', Percentage(), ' '], maxval=len(l)).start()
c = 0
for e in l:
g = bfu.vec2g(e, n)
g = bfu.vec2g(e,n)
if not gk.scc_unreachable(g):
# s = trv.edge_backtrack2g1_directed(g2, capsize=CAPSIZE)
s = trv.v2g22g1(g, capsize=CAPSIZE, verbose=False)
s = ur.liteqclass(g, capsize=CAPSIZE, verbose=False)
else:
s = set()
if s:
@@ -96,9 +95,9 @@ def find_nearest_reachable(g2, max_depth=4):
step += 1


def wrapper(fold, n=10, dens=0.1):
def wrapper(fold, n=10, dens=0.1, urate=URATE):
scipy.random.seed()
rate = 2
rate = urate

r = None
s = set()
@@ -130,9 +129,9 @@ def wrapper(fold, n=10, dens=0.1):
# raise ValueError
startTime = int(round(time.time() * 1000))
if EST == 'pc':
g2 = pc.dpc(data[:, ::2], pval=0.0001)
g2 = pc.dpc(data[:, ::rate], pval=0.0001)
elif EST == 'svar':
g2 = lm.data2graph(data[:, ::2])
g2 = lm.data2graph(data[:, ::rate])
if trv.density(g2) < 0.7:
print gk.OCE(g2, true_g2)
# s = examine_bidirected_flips(g2, depth=DEPTH)
@@ -198,18 +197,19 @@ def wrapgen(fold, n=10, dens=0.1):

print 'processes: ', PNUM, INPNUM

densities = {6: [0.25, 0.3, 0.35],
densities = {5: [0.25, 0.3, 0.35],
6: [0.25, 0.3, 0.35],
8: [.15, .2, 0.25, 0.3],
10: [0.3],
15: [0.1],
20: [0.1],
25: [0.1],
30: [0.1],
35: [0.1]}
10:[0.3],
15:[0.1],
20:[0.1],
25:[0.1],
30:[0.1],
35:[0.1]}

wrp = wrapper

for nodes in [10]:
for nodes in [6]:
z = {}
pool = Pool(processes=PNUM)
for dens in densities[nodes]:
42 changes: 21 additions & 21 deletions gunfolds/tools/bfutils.py
Original file line number Diff line number Diff line change
@@ -20,6 +20,18 @@ def isSclique(G):
return True


def pure_directed_inc(G, D):
G_un = {}
# directed edges
for v in D:
G_un[v] = {}
for w in D[v]:
if G[w]:
for e in G[w]:
G_un[v][e] = set([(0, 1)])
return G_un


def directed_inc(G, D):
G_un = {}
# directed edges
@@ -46,26 +58,6 @@ def bidirected_inc(G, D):
return G


def increment_u(G_star, G_u):
# directed edges
G_un = directed_inc(G_star, G_u)
# bidirected edges
G_un = bidirected_inc(G_un, G_u)
return G_un


def pure_directed_inc(G, D):
G_un = {}
# directed edges
for v in D:
G_un[v] = {}
for w in D[v]:
if G[w]:
for e in G[w]:
G_un[v][e] = set([(0, 1)])
return G_un


def increment(g):
'''
undersample g by 2
@@ -101,6 +93,14 @@ def dincrement_u(G_star, G_u):
return G_un


def increment_u(G_star, G_u):
# directed edges
G_un = directed_inc(G_star, G_u)
# bidirected edges
G_un = bidirected_inc(G_un, G_u)
return G_un


def undersample(G, u):
Gu = G
for i in range(u):
@@ -350,7 +350,7 @@ def iall(i, nodes):
def cc_all(i, nodes, steps):
# print i
g = num2CG(i, nodes)
return cc_undersamples(g, steps=steps)
return i, cc_undersamples(g, steps=steps)


def make_rect(l):
6 changes: 3 additions & 3 deletions gunfolds/tools/calc_procs.py
Original file line number Diff line number Diff line change
@@ -6,16 +6,16 @@ def get_process_count(starts):
""" Using the number of starts per graph and machine info, calculate
the number of processes to spawn """
if socket.gethostname().split('.')[0] == 'leibnitz':
num_processes = 30
num_processes = 60
num_processes = max((1, num_processes / starts))
elif socket.gethostname().split('.')[0] == 'mars':
num_processes = 21
num_processes = 12
num_processes = max((1, num_processes / starts))
elif socket.gethostname().split('.')[0] == 'saturn':
num_processes = 12
num_processes = max((1, num_processes / starts))
elif socket.gethostname().split('.')[0] == 'hooke':
num_processes = 22
num_processes = 21
num_processes = max((1, num_processes / starts))
else:
# Setting the number of parallel running processes to the number
14 changes: 8 additions & 6 deletions gunfolds/tools/graphkit.py
Original file line number Diff line number Diff line change
@@ -119,11 +119,13 @@ def scale_free(n, alpha=0.7, beta=0.25,
return g


def ring(n):
def ring(n, permute=False):
g = {}
for i in range(1, n):
g[str(i)] = {str(i + 1): set([(0, 1)])}
g[str(n)] = {'1': set([(0, 1)])}
names = [str(x+1) for x in range(n)]
if permute: names = np.random.permutation(names)
for i in range(n-1):
g[names[i]] = {names[i+1]: set([(0,1)])}
g[names[n-1]] = {names[0]: set([(0,1)])}
return g


@@ -158,8 +160,8 @@ def ringarcs(g, n):
return g


def ringmore(n, m):
return ringarcs(ring(n), m)
def ringmore(n, m, permute=False):
return ringarcs(ring(n,permute=permute), m)


def digonly(H):
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.