NAS

NAS performance with kernel 3.19.0

7 14304
tkaiser  
Edited by tkaiser at Wed Feb 11, 2015 03:43

Based on Igor's image build scripts I started with Kernel 3.19.0-rc5 and Debian Wheezy. Currently the cpufreq stuff isn't working in 3.19 and there's a lot of related work pending or will be available later with kernel 3.20. Since I tried to maximize performance I tried a few things, especially clock the CPU cores as high as possible (the default cpufreq in mainline is just 912 MHz!)

From Igor's build directory I patched output/u-boot/include/configs/sun7i.h from 912 MHz to 1056 MHz (everything above failed in my limited tests). Setting the cpufreq in u-boot is currently the only method supported:
  1. #define CONFIG_CLK_FULL_SPEED                1056000000
Copy the Code
And I modified the kernel config since it is known that some settings positively influence SMP performance on ARM (compare with http://forum.lemaker.org/thread-7102-1-1.html). So I set in lib/config/linux-sunxi-next.config:
  1. CONFIG_SCHED_MC=y
  2. CONFIG_SCHED_SMT=y
Copy the Code
(CONFIG_HZ_100=y and CONFIG_HZ=100 were already set before).

Isolated network tests / iperf throughput:

Tests without further tuning (only eth0 IRQs handled by CPU1 and slight 'overclocking' to 1056MHz...

OS X --> BPi:
  1. root@bananapi:~# iperf -s
  2. ------------------------------------------------------------
  3. Server listening on TCP port 5001
  4. TCP window size: 85.3 KByte (default)
  5. ------------------------------------------------------------
  6. [  4] local 192.168.83.44 port 5001 connected with 192.168.83.247 port 63446
  7. [ ID] Interval       Transfer     Bandwidth
  8. [  4]  0.0-10.0 sec  1.09 GBytes   934 Mbits/sec
  9. [  5] local 192.168.83.44 port 5001 connected with 192.168.83.247 port 63447
  10. [  5]  0.0-10.0 sec  1.10 GBytes   940 Mbits/sec
  11. [  4] local 192.168.83.44 port 5001 connected with 192.168.83.247 port 63448
  12. [  4]  0.0-10.0 sec  1.10 GBytes   940 Mbits/sec
  13. [  5] local 192.168.83.44 port 5001 connected with 192.168.83.247 port 63449
  14. [  5]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
  15. [  4] local 192.168.83.44 port 5001 connected with 192.168.83.247 port 63450
  16. [  4]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec
Copy the Code
BPi --> OS X:
  1. mountain-mini:~ admin$ iperf -s
  2. ------------------------------------------------------------
  3. Server listening on TCP port 5001
  4. TCP window size:  128 KByte (default)
  5. ------------------------------------------------------------
  6. [  4] local 192.168.83.247 port 5001 connected with 192.168.83.44 port 45648
  7. [ ID] Interval       Transfer     Bandwidth
  8. [  4]  0.0-10.0 sec   689 MBytes   577 Mbits/sec
  9. [  4] local 192.168.83.247 port 5001 connected with 192.168.83.44 port 45649
  10. [  4]  0.0-10.0 sec   800 MBytes   671 Mbits/sec
  11. [  4] local 192.168.83.247 port 5001 connected with 192.168.83.44 port 45650
  12. [  4]  0.0-10.0 sec   803 MBytes   673 Mbits/sec
  13. [  4] local 192.168.83.247 port 5001 connected with 192.168.83.44 port 45651
  14. [  4]  0.0-10.0 sec   806 MBytes   676 Mbits/sec
  15. [  4] local 192.168.83.247 port 5001 connected with 192.168.83.44 port 45652
  16. [  4]  0.0-10.0 sec   798 MBytes   669 Mbits/sec
Copy the Code
SATA / iozone results:

In 3.19 it's no longer necessary to patch sunxi_ahci to work either with or without a port multiplier.

EDIT: That's not true. I had to patch the kernel (unset AHCI_HFLAG_NO_PMP) and unfortunately my JMB321 seems to not like my SSD that much. Regardless on which PM port I connect the SSD the negotiation will always just be SATA 1.0 (1.5 Gbps). That's also the explanation for the decreased read throughput.

I tested with a JMB321 in between Banana and the EVO 840:
  1.      KB  reclen   write rewrite    read    reread
  2. 2048000       4   42281   42694   137476   137470
  3. 2048000      32   42025   42261   137427   137461
  4. 2048000     512   41590   41938   135917   136992
  5. 2048000   16384   41495   42008   136863   136903
Copy the Code
It makes no difference whether CONFIG_SCHED_MC=y and CONFIG_SCHED_SMT=y are set or not. Different situation when there's no PM as bottleneck:

"CONFIG_SCHED_MC not set" and CONFIG_SCHED_SMT not set":
  1.      KB  reclen   write rewrite    read    reread
  2. 2048000       4   43053   43643   202636   203209
  3. 2048000      32   42743   42949   184831   183612
  4. 2048000     512   41774   42533   174186   175606                                                                          
  5. 2048000   16384   41897   42500   175786   172778
Copy the Code
CONFIG_SCHED_MC=y and CONFIG_SCHED_SMT=y:
  1.      KB  reclen   write rewrite    read    reread
  2. 2048000       4   44413   44092   212359   212756
  3. 2048000      32   44078   44115   191354   192068
  4. 2048000     512   44167   43707   184329   184778
  5. 2048000   16384   43238   43798   185875   186162
Copy the Code
Combined tests / LanTest results:



The tweaked kernel settings seem to provide another few MB/sec due to higher SATA throughput. Interestingly the situation changes when one uses the '10 GBit Ethernet settings' (larger test files and 1024K blocksize instead of 128K as before):



Looks promising especially when you have in mind that it will be possible to clock the BPi with 1200 MHz again sooner or later (and then throughput will increase again a little blt).
tkaiser  
Some final words regarding this test: I removed the PM again and let Igor's image install on the SSD (/root/nand-sata-install.sh) and applied some network tuning:
  1. sysctl -w net/core/rmem_max=8738000
  2. sysctl -w net/core/wmem_max=6553600
  3. sysctl -w net/ipv4/tcp_rmem="8192 873800 8738000"
  4. sysctl -w net/ipv4/tcp_wmem="4096 655360 6553600"
  5. sysctl -w vm/min_free_kbytes=65536
  6. sysctl -w net.ipv4.tcp_window_scaling=1
  7. sysctl -w net.ipv4.tcp_timestamps=1
  8. sysctl -w net.ipv4.tcp_sack=1
  9. sysctl -w net.ipv4.tcp_no_metrics_save=1
Copy the Code
And I assigned the 3 processes serving my Mac (cnid_metad, cnid_dbd and the afpd serving my session) to CPU 0 (if you would use Windows/Samba this would apply to the smbd process serving your session instead):
  1. for i in 2488 2494 2495 ; do taskset -p 01 $i ; done
Copy the Code
Now it looks like this (still with just 1056 MHz):



And with 3000M filesize and 1024K blocksize:



The measured values stayed pretty stable during all 5 test runs. Some explanations to the LanTest parameters as well as their relation to 'real world scenarios' can be found here: http://www.helios.de/viewart-de.html?id=1711

tkaiser  
Edited by tkaiser at Mon Feb 9, 2015 05:52

I made another test with Igor's build scripts, 3.19.0-rc6, different kernel sources (using Maxime Ripard's sunxi-next branch instead of 'official' mainline), slightly different kernel config, different u-boot (using mainline instead of Robert C. Nelson's) but still 1056Mhz instead of the default 912Mhz.

SATA throughput measured with iozone is a bit slower and this also applies to network throughput measured with iperf after the usual TCP/IP tuning outlined above:



The LanTest results look ok but the orange triangles indicate that there's something wrong (sequential writes between 24.4 MB/s min. and 38.2 MB/s max. --> too much difference) but it's too early to do serious benchmarking because I changed another thing too: the board itself



(it's a LinkSprite pcDuino3 Nano comparable to the original Banana Pi. I applied Adam Sampson's patches to u-boot and kernel sources and modified Igor's scripts a bit. Two problems remain: currently broken mainline u-boot not able to read/parse boot.scr and maybe the DRAM clock frequency that Adam set to 408 -- will try with 480 later)

BTW: The pcDuino's SATA power connector is compatible with the cubie's but not the Pi. So SATA/power cables made for pcDuino or cubieboards must not be used with Banana/Orange Pi and vice versa!

tkaiser  
Hmm... I increased DRAM clock to 480: http://pastebin.com/kCusGxQm

And while it seems to be stable unfortunately the network throughput doesn't increase (compared to the BPi it's way slower). I've done some standard iperf tests:

OS X --> pcDuino
  1. [  4]  0.0-10.0 sec   570 MBytes   478 Mbits/sec
  2. [  4] local 192.168.83.74 port 5001 connected with 192.168.83.73 port 45115
  3. [  4]  0.0-10.0 sec   594 MBytes   498 Mbits/sec
  4. [  4] local 192.168.83.74 port 5001 connected with 192.168.83.73 port 45116
  5. [  4]  0.0-10.0 sec   840 MBytes   704 Mbits/sec
  6. [  4] local 192.168.83.74 port 5001 connected with 192.168.83.73 port 45117
  7. [  4]  0.0-10.0 sec   611 MBytes   512 Mbits/sec
  8. [  4] local 192.168.83.74 port 5001 connected with 192.168.83.73 port 45118
  9. [  4]  0.0-10.0 sec   604 MBytes   506 Mbits/sec
  10. [  4] local 192.168.83.74 port 5001 connected with 192.168.83.73 port 45119
  11. [  4]  0.0-10.0 sec   716 MBytes   600 Mbits/sec
Copy the Code
pcDuino --> OS X
  1. [  4]  0.0-10.0 sec   763 MBytes   640 Mbits/sec
  2. [  5] local 192.168.83.73 port 5001 connected with 192.168.83.74 port 49502
  3. [  5]  0.0-10.0 sec   389 MBytes   326 Mbits/sec
  4. [  4] local 192.168.83.73 port 5001 connected with 192.168.83.74 port 49503
  5. [  4]  0.0-10.0 sec   400 MBytes   336 Mbits/sec
  6. [  5] local 192.168.83.73 port 5001 connected with 192.168.83.74 port 49505
  7. [  5]  0.0-10.0 sec   397 MBytes   333 Mbits/sec
  8. [  4] local 192.168.83.73 port 5001 connected with 192.168.83.74 port 49507
  9. [  4]  0.0-10.0 sec   834 MBytes   699 Mbits/sec
  10. [  5] local 192.168.83.73 port 5001 connected with 192.168.83.74 port 49509
  11. [  5]  0.0-10.0 sec   788 MBytes   661 Mbits/sec
Copy the Code
And did two times iozone tests on the SATA SSD (using "iozone -a -g 2000m -s 2000m -i 0 -i 1 -r${recordsize}k" with different record sizes):
  1. 2048000       4   41588   47520   218494   219156
  2. 2048000       4   41247   48200   218171   219174
  3. 2048000      32   40930   48094   196609   197547
  4. 2048000      32   41417   47526   198031   198522
  5. 2048000     512   40384   47232   192915   192401
  6. 2048000     512   42114   46416   192788   193241
  7. 2048000   16384   46025   44079   195094   195918
  8. 2048000   16384   43941   46504   222510   223501
Copy the Code
LanTest results look good except of write performance. Here the network throughput seems to be the bottleneck:

hmmm nice results

tkaiser  
rutierut replied at Mon Feb 9, 2015 05:12
hmmm nice results

Unfortunately partly wrong or let's better say: not comparable directly. I made another set of iperf tests with the same Mac I used before to test against Banana Pi and was also able to get ~940 Mbits/sec in one and above 750 Mbits/sec in the other direction. So I have to further investigate what's wrong here, since I always get slower results with a machine connected directly to the same switch (HP 1810-8G v2) as the BPi/pcDuino compared to the superiour results the other machine achieves that is three hops (switches) away.

Another interesting thing was to measure the influence of DRAM clock. I tested the pcDuino with both 408MHz DRAM clock as well as 480MHz. Always two times using "iozone -a -g 2000m -s 2000m -i 0 -i 1 -r${recordsize}k":

With 480 Mhz DRAM and 1056MHz cpufreq:
  1. 2048000       4   41588   47520   218494   219156
  2. 2048000       4   41247   48200   218171   219174
  3. 2048000      32   40930   48094   196609   197547
  4. 2048000      32   41417   47526   198031   198522
  5. 2048000     512   40384   47232   192915   192401
  6. 2048000     512   42114   46416   192788   193241
  7. 2048000   16384   46025   44079   195094   195918
  8. 2048000   16384   43941   46504   222510   223501
Copy the Code
With just 408 Mhz DRAM and also 1056MHz cpufreq:
  1. 2048000       4   37368   43087   206034   208048
  2. 2048000       4   41708   43635   207130   207796
  3. 2048000      32   42540   42384   180647   179998
  4. 2048000      32   40230   42918   183590   184689
  5. 2048000     512   42601   42131   175569   175094
  6. 2048000     512   40822   43882   177408   176607
  7. 2048000   16384   38623   43127   203863   203676
  8. 2048000   16384   43976   44185   178184   178290
Copy the Code
But since figuring out at which frequency the DRAM still works reliable requires a lot of efforts I would stay with the manufacturer's defaults (408MHz on the pcDuino3 nano, 432MHz on Banana Pi/Pro)

http://pipelettes-au-jardin.xooi ... wprofile&u=3317
http://26403.dynamicboard.de/u88494_yehwodi.html
http://ajt.xooit.org/profile.php?mode=viewprofile&u=4266
http://communities.curl.com/memb ... ofile&uid=11855
https://www.zspk.gov.ru/forum/me ... t=reply#message1331
https://forum.pcformat.pl/yehwodi-u
https://forum.ct8.pl/member.php?action=profile&uid=17260
https://forum.thirtybees.com/user/samdesign
http://googl.wc.lt/user.php?login=samdesign&view=upvoted
http://treehouse.ofb.net/go/en/user/23683
http://pattern-wiki.org/wiki/User:Samdesign
https://forum.ark.io/user/samdesign
http://www.bullypedia.net/forums ... rofile&uid=4855
https://www.mobafire.com/profile/samdesign-839571
http://agropoisk.ru/forum/profil ... 56349881a7ad0c988f2
http://tale-of-tales.com/forum/p ... profile&u=41161
http://komjatice.fara.sk/profile.php?lookup=1888
http://blog.avtosreda.ru/blogs/2 ... e.html#comment-9790
http://forums.defiance.com/showt ... ode-7&p=2119171
https://forum.techland.pl/profil ... ield_core_pfield_11
http://www.consolegames.ro/forum/members/yehwodi/
https://www.engagemedia.org/author/samdesign
http://soulthirteen.xoo.it/profile.php?mode=viewprofile&u=189
http://teenyland.siteboard.eu/u553_samdesign.html
http://asa.forum24.se/asa-post-13975.html#13975
http://106snowscoot.clicforum.co ... armeacut.htm#p96514
https://evermotion.org/vbulletin/member.php?887009-yehwodi
http://knsz.prz.edu.pl/forum/mem ... file&uid=101231
http://a-d-e.vraiforum.com/profile.php?mode=viewprofile&u=929
https://forum.zenstudios.com/member.php?2302739-yehwodi
http://kbforum.dragondoor.com/members/yehwodi.html
https://forums.jetphotos.com/member.php?32429-samdesign
http://lpbc.thischurch.org/mod/discuss/view.php?cat_id=9671
http://aowdbbs.snail.com/en/view ... 84&p=4215#p4215
https://forum.avscripts.net/member.php?28421-yehwodi
https://forum.pjrc.com/members/59275-yehwodi
http://archeslocal.org.uk/forum/ ... file&uid=100847
https://www.ideafit.com/answers/ ... eting-and-retention
http://210782.homepagemodules.de/u145574_samdesign.html
http://msmjapan.xooit.com/profile.php?mode=viewprofile&u=1180
https://gtranslate.io/forum/member/yehwodi/
https://www.producthunt.com/@samdhprint
https://www.behance.net/samdhprint
http://talk-staging.plesk.com/members/yehwodi.183876/
https://forum.tp-link.com/member.php?642286-yehwodi
http://compbio.clemson.edu/forum ... cfad845095cfacd9abd
https://forums.macrumors.com/members/rotary-12.1141609/
http://www.seomastering.com/seo- ... ea=summary;u=126360
https://discussion.evernote.com/ ... ield_core_pfield_12
http://kraftymarketingprofits.co ... m/user-5256-yehwodi
https://www.question2answer.org/qa/user/samdhprint
http://chryslerclub.su/forum/mem ... wprofile&u=6800
https://rockethub.com/member/577117
https://dev.to/samdhprint
https://triberr.com/samdhprint
https://www.addpoll.com/samdesign
http://investimonials.com/users/samdhprintmail@gmail.com.aspx
https://www.colourlovers.com/lover/yehwodi
http://noxiousroleplay.nn.pe/mem ... rofile&uid=2247
http://www.pbase.com/samdesign/profile
http://www.craftfoxes.com/profiles/yehwodi
https://mygeohub.org/members/632 ... is-digital-printing
https://devpost.com/samdhprintma ... f_medium=global-nav
https://pastebin.com/u/samdhprint
https://trello.com/samdhprint
https://www.mixcloud.com/samdhprint/
https://www.trover.com/u/samdhprint
https://issuu.com/samdhprint
https://www.caclubindia.com/profile.asp?member_id=2505628
https://www.ukbusinessforums.co.uk/members/sami-designer.311314/
https://forum.lowyat.net/index.php?showuser=1037388
https://forum.polymake.org/membe ... a052abc11a32d91ca95
http://www.vcfed.org/forum/member.php?46173-samdesigner
http://rhizome.org/profile/sam-design/
https://www.amazon.de/gp/profile ... ef=hud_2_gw_profile
https://www.openstreetmap.org/user/samdesignsam
http://lib.akb.nis.edu.kz/user/samdesign/
https://www.eventbrite.it/o/samdhprint-17625999325
https://www.intensedebate.com/people/samdesign
https://www.l-camera-forum.com/user/60899-samdesign/
https://www.slideshare.net/samidesign
https://www.ifixit.com/User/2615 ... +%D8%B3%D8%A7%D9%85
http://www.basenotes.net/members/26245574-yehwodi
https://www.woltlab.com/user/1494969-yehwodi/
http://www.dead.net/member/samdesigner
https://www.tapatalk.com/groups/ ... ewprofile&u=354
http://www.siteownersforums.com/member.php?u=183652
https://www.theverge.com/users/samdesign
https://www.zotero.org/yehwodi
http://www.domoticz.com/forum/me ... profile&u=20861
https://glosbe.com/profile/64213 ... ub-3314662261394816
http://www.imfaceplate.com/samdhprint/what-is-digital-printing
http://www.imfaceplate.com/samdhprint/what-is-a-poster-printing
http://www.imfaceplate.com/samdhprint/what-is-a-leaflet-printing
http://www.imfaceplate.com/samdh ... and-all-sorts-of-it
http://www.imfaceplate.com/samdhprint/what-is-a-printing-catalog
http://www.imfaceplate.com/samdh ... nting-and-its-types
http://www.imfaceplate.com/samdh ... -banner-printing-is
https://www.yates.co.nz/garden-club/public/107478
https://forum.cs-cart.com/user/35651-samdesign/
http://proline.physics.iisc.erne ... .php/User:Samdesign
http://hullcity.boardhost.com/viewtopic.php?pid=376675#p376675
http://www.domoticz.com/forum/me ... profile&u=20861
https://lichess.org/@/yehwodi
https://knoji.com/meet/samdhprint/
https://www.misterpoll.com/users/455285
http://heisenberg.csic.edu.uy/yehwodi
http://microfluidics.utoronto.ca/gitlab/yehwodi
https://www.eurekavideo.co.uk/users/yehwodi
http://www.magcloud.com/user/yehwodi
http://projects.umwhistory.org/cwh/myomeka/posters/show/1949
https://www.viki.com/users/samdhprint
http://uid.me/samdhprint
https://cookpad.com/id/pengguna/12813022
http://myfolio.com/yehwodi
https://www.flicks.co.nz/member/yehwodi/
http://www.tribunnews.com/member/yehwodi
https://bible.org/users/yehwodi
http://cloud.ld.ttu.ee/idu0110/U ... /76479/Default.aspx
http://orcadmarketplace.com/User ... /32484/Default.aspx
http://danmooredesigns.com/UserP ... 220678/Default.aspx
http://holocaustmusic.org/UserPr ... 657478/Default.aspx
https://www.redbubble.com/people ... -dropdown&asc=u
https://www.burdastyle.com/profiles/yehwodi/my_activity
https://everplaces.com/yehwodi
http://www.tripntale.com/profile/118877
http://zakupki.gov.ru/forum/user/profile/1622840.page
https://xxm.times.uh.edu/forums/users/yehwodi
https://www.fontshop.com/people/samdhprint
https://hackerone.com/samdhprint ... mdhprint&page=1
https://torgi.gov.ru/forum/user/profile/541834.page
https://sol.1c.ru/forum.php?PAGE_NAME=profile_view&UID=19521
https://forum.teamspeak.com/members/378038-yehwodi
http://tseng.s2.bizhat.com/tseng-post-234178.html#234178
https://www.goodreads.com/user/show/82856669-samdhprint
http://www.rohitab.com/discuss/user/70512-yehwodi/
http://www.cplusplus.com/user/profile.cgi?w=show
https://qelectrotech.org/forum/profile.php?id=3309
https://www.raspberrypi.org/foru ... 2b235a0754bb3df771c
http://wizaz.pl/forum/member.php ... 7661#vmessage297661
https://speakerdeck.com/yehwodi
https://www.quicktopic.com/52/H/wV5EZJj8mvSu
http://forum.imagespaceinc.com/member.php?23617-yehwodi
http://www.designerstalk.com/for ... -of-companies?new=1
http://forum1.shellmo.org/member ... file&uid=194118
https://www.mybbarab.com/user-230008.html
https://web.poe.garena.tw/forum/ ... 81/page/16#p1718321
http://www.unicyclist.com/forums/member.php?u=139632
http://siteownersforums.com/member.php?u=183652
http://www.lawyersclubindia.com/profile.asp?member_id=427009
http://forum.deasoft.com/member.php?u=297977
https://forum.wpde.org/members/yaldaa.108247/
http://www.fitmisc.net/forum/member.php?11054-yaldaa
https://armorgames.com/user/yehwodi
https://coub.com/samdhprint
http://www.folkd.com/user/yehwodi
https://www.siasat.pk/forums/members/yaldaa.205522/
https://forum.homag.com/member/522-maryamdolati/about
http://cssa.com/2015n/en/forum/profile/2923-yehwodi
http://www.dbtools.com.br/EN/for ... hp?thread_id=331993
http://arahn.100webspace.net/viewtopic.php?p=740826#740826
https://l2e-global.com/forum/members/maryamdolati.51260/#about
https://www.bolsovercruiseclub.c ... ield_core_pfield_11
https://forums.autodesk.com/t5/a ... light/false#M697396
https://emirate-dubai.locanto.ae ... gital-printing.html
http://ttlink.com/samdesign/detail
https://www.instapaper.com/p/6630034
http://3cjq08j-dukj3ikar.mycindr.com/
http://www.im-creator.com/free/samdesign/samdhprint
https://steepster.com/samdhprint
http://samdhprint.simpsite.nl/
https://kinja.com/samdhprints
https://speakerdeck.com/samidesigner
https://devpost.com/cashbussieness
https://buddypress.org/members/samidesigner/profile/
http://samdhprint.moonfruit.com/
https://sami-bohner.webnode.com/
https://www.codecademy.com/samidesigner
http://samdhprint.jigsy.com/
http://www.123formbuilder.com/form-3998299/My-Form
http://samdhprints-first-project-0c303a.webflow.io/
http://www.i-m.mx/samdhprint/samdhprint/
https://www.programmableweb.com/profile/samdhprint
https://printingsam.blogspot.com/
https://samdhprint.kinja.com/1827284367
https://infogram.com/digitalprinting-1h7z2l0o0o8y2ow
http://samdhprint.blog.fc2.com/
https://khnh-trhn-sm.jimdosite.com/
https://portfoliothought.com/for ... rofile&uid=4965
http://www.wrightbrotherscustoms ... 0acb6c380b391a87fd6
http://www.nfomedia.com/profile?uid=rNeZZj
http://forums.devarticles.com/member.php?u=637320
https://www.vocabulary.com/profiles/A1KACT80PK3ESH
https://www.avitop.com/cs/members/yehwodi.aspx
http://www.feedbooks.com/user/4429942/profile
http://www.seobook.com/user/yehwodi
http://tehran.academia.edu/samdhprint
https://about.me/samdhprint
https://knowyourmeme.com/users/sam-dhprint
https://web.bricksite.net/8duf/jun-25/
http://digitalprinting.simplesite.com/
http://samdhprint.wallinside.com/
https://profiles.wordpress.org/yehwodi/
http://samdhprint.e-monsite.com/
http://samdhprint.eklablog.com/
https://samdhprint.page.tl/
http://publish.lycos.com/samdhpr ... %D8%A8%D9%86%D8%B1/
https://padlet.com/samdhprintmail/samdhprint
http://samdhprint.strikingly.com/
http://samdhprint.over-blog.com/
http://samdhprint.wikidot.com/
http://groupspaces.com/samdhprint
https://sites.google.com/view/samdhprint/home
http://samdhprint.zohosites.eu/
https://samdhprint.webnode.com/
https://www.quora.com/profile/Sam-Designers-House
http://samdhprint.bravesites.com/
https://samdhprint.livejournal.com/
http://samdhprint.yolasite.com/
https://samdhprintmail.wixsite.com/samdhprint
https://samdhprint.hatenablog.com/
https://samdhprint.weebly.com/
http://samdhprint.doodlekit.com/
https://samdhprint.wordpress.com/
http://samdhprint.webs.com/
https://samdhprint.site123.me/
https://www.evernote.com/shard/s ... bd92119c0a085f1dc7b
http://samdhprint.webstarts.com/
https://samdhprint-54.webself.net/
http://guildwork.com/users/yehwodi
https://torgi.gov.ru/forum/user/profile/541834.page
https://www.studiopress.com/forums/users/yehwodi/
https://social.microsoft.com/Profile/samdhprint
http://bbs.vernee.cc/home.php?mod=space&uid=519761
https://500px.com/samdhprint
http://pixelhub.me/samdhprint
https://steepster.com/samdesign
https://app.hiive.co.uk/profile/ ... 596-6d39673ef671/#/
https://sketchfab.com/samdhprint
https://members.nationalgeographic.com/480619011818/
https://mootools.net/forge/profile/samdesign
https://gitlab.labs.nic.cz/samdesign
https://www.patreon.com/samdhprint/creators
http://firebaptizedholiness.ning.com/profile/samdesign
http://whazzup-u.com/profile/samdhprint
https://creativemarket.com/samdesignsam
https://growthhackers.com/members/_6536633607054
https://seekingalpha.com/user/49627916/followers
https://hub.docker.com/u/samdesignsam/
http://forum.eagleget.com/profile.php?id=4448
http://www.joomlatune.com/forum/ ... ;u=32299;sa=summary
https://form.jotformeu.com/81642023911347
https://www.homify.co.uk/ideaboo ... -a-digital-printing
https://community.zoiper.com/user/samdesign
http://vator.tv/person/sam-designers-house
https://connect.sensiolabs.com/profile/samdesign
https://subrion.org/members/info/yehwodi/
https://samdhprint.contently.com/
http://organisationalamaison.sof ... initial.htm#p229639
http://tmkt.forogratis.es/post42248.html#p42248
http://forum.powerbox-systems.com/index.php?user/4080-yehwodi
http://vseobovsem.getbb.ru/membe ... ewprofile&u=652
http://forum.odin.com/members/daniellll.192926/
http://www.effecthub.com/user/1554169
http://jsdo.it/samdhprint
https://www.forexfactory.com/yehwodi
http://z13.invisionfree.com/Writ ... ex.php?showuser=415
https://www.freemathhelp.com/forum/members/68064-yehwodi
http://www.myvidster.com/profile/samdesign
https://remote.com/khnhTrHn-sm
https://www.inprnt.com/profile/yehwodi/
https://forums.prosportsdaily.com/member.php?491428-samdesign
https://gitlab.ite.tul.cz/u/samdesign
https://olioboard.com/users/samdesign
https://www.edocr.com/user/samdhprint
https://public.tableau.com/profile/samdesign#!/
https://www.chordie.com/forum/profile.php?id=875538
https://musicbrainz.org/user/samdesign
https://forums.createspace.com/e ... design?view=profile
https://community.articulate.com/users/samdesign
http://www.authorstream.com/samdesign/
https://www.f6s.com/samdesignershouse
https://telegra.ph/%DA%86%D8%A7% ... 8%D9%86%D8%B1-09-16
http://kvircfr.les-forums.com/user/2417/
http://kirovnet.ru/forum/index.php?action=profile
https://www.hiveworkshop.com/members/samdesign.269655/
https://www.apoi.ru/profile/?id=84428
http://www.apoi.ru/forum/memberl ... e4bd1359dde9a9798cd
https://php.ru/forum/members/samdesign.69914/
https://www.takbook.com/forum/members/yaldaa/
http://foodtube.net/profile/samdesign
http://www.flashflashrevolution.com/profile/samdesign/
https://nolamotor.com/NOLAforum/ ... rofile&uid=6942
http://www.php-bb.ir/forum/yaldaa-u1573/
http://www.artiloo.com/phpBB2/pr ... wprofile&u=5051
https://scioly.org/forums/member ... profile&u=39166
http://vstarcam.com/bbs/home.php?mod=space&uid=42288
http://www.laborlawtalk.com/member/717246-samdesign
https://christliche-teestube.for ... wprofile&u=4037
http://panacea-bocaf.org/forum/m ... 350d19754ae7e3455a4
http://www.cogsci.nl/forum/profile/samdesign
https://www.craiovaforum.ro/users/samdesign/
https://talkwebber.ru/members/samdesign.83832/
http://www.forum.atheism.ru/memb ... profile&u=51887
http://forum.p-stats.com/members/112614-samdesign
http://www.ultidev.com/forums/yaf_profile25453.aspx
http://vrn.best-city.ru/users/samdesign/
https://forum.bakililar.az/profile/79903-samdesign/
http://forum.hugogames.com/member.php?2713-samdesign
https://forum.uti-puti.com.ua/me ... profile&u=23280
http://kiansat.kim/member.php?u=248012
http://forum.roditeli.ua/memberl ... profile&u=67814
https://www.digitarahan.com/forum/index.php?members/yaldaa.165
http://marocano.cforum.info/prof ... wprofile&u=3332
http://forum.keden.kz/memberlist.php?mode=viewprofile&u=46154
https://forum.studio-397.com/index.php?members/samdesign.51314/
https://www.aswaqcity.com/members/364568/
http://king-kong.fansforum.info/ ... ewprofile&u=204
http://a-vos-etriers.discuforum. ... ewprofile&u=501
https://www.xossip.com/member.php?u=2305512
http://www.edu4pal.koom.ma/vb/member.php?u=1334
https://groupmoney.ru/profile/8783-samdesign/
http://player.ru/member.php?u=153863
https://kasperskyclub.com/samdesign
http://alhoush.com/sam-design/
http://alhoush.com/forums/member.php?264829-samdesign
http://www.separateddads.co.uk/forum/user-55213.html
http://primmarketing.ru/forum/me ... ff5b470771c76b7d751
https://forum.yugiohcardmaker.net/user/906926-samdesign/
https://curecoin.net/forum/profile/samdesign/
http://forum.aa-slovenia.si/memb ... 1c03376b008f599ba10
http://forum.vlz.ru/memberlist.php?mode=viewprofile&u=60433
http://www.escalade-alsace.com/f ... wprofile&u=1399
http://www.irosclass.ir/yaldaa/
http://alinez-hacker.mwik.com/pr ... wprofile&u=1124
http://alaroob.com/forums/member.php?227118-samdesign
http://mobilestore.coolforum.inf ... wprofile&u=6106
http://www.disputt.no/member/256310-samdesign
https://www.evernote.com/shard/s ... 6c72242c962199ad979
https://lightboxsam.quora.com
https://lightbax.blogspot.com/
https://sites.google.com/site/lightbax/
http://www.123formbuilder.com/form-4258406/-
http://forum.dlfox.com/members/yaldaa.html
https://forums.socialpointgames.com/user/samdesign
http://www.mihangame.com/forum/member.php?u=4405
https://forum.hardedge.org/index.php?user/5906-samdesign/
http://forum.iact.ir/member.php?action=profile&uid=539
https://www.voipiran.io/forums/profile/15258-yaldaa
https://forum-zarabotok.ru/profile/13158-samdesign
http://lost-places.com/forum/index.php/User/6449-samdesign/
http://forum.mobology.ir/member.php?u=2090
https://withthewill.net/members/10967-samdesign
http://www.roozmenu.com/member.php?u=26101
http://forum.activerayan.ir/member.php?122-yaldaa
http://www.sbirme.org/modules/profile/userinfo.php?uid=899743
http://wu-world.com/profiles/blogs/how-to-digital-printing
https://forum.irani-dl.com/member4602
https://3drpilots.com/members/samdesign.15575/
http://www.aerocenter.ir/forum/members/44830.html
http://forum.glogah.org/Thread-% ... C%D8%B3%D8%AA%D8%9F
https://forum.s15oc.com/members/6354.html
http://nerdforum.org/index.php?/user/3190-samdesign/
http://www.solarin.ir/forum.php/ ... tle//page/1#post199
http://www.maprecord.com/forum/m ... ofile&uid=11877
http://www.ottawavalleyairsoft.ca/forums/user-6254.html
http://forum.irmug.com/viewtopic ... &p=55919#p55919
https://answercenter.ir/profile/10976-yaldaa/
https://www.robot-forum.com/robotforum/index.php?action=profile
http://homestudioitalia.com/forum/user/47784-samdesign/
http://wbb.travianforum.ir/user/101623-samdesign
http://moms-diner.com/recipes/users/profile/samdesign
https://chocolatey.org/profiles/samdesign
https://www.openstreetmap.org/user/samdesignsam
https://community.articulate.com/users/samdesign
https://forums.createspace.com/en/community/people/samdesign
https://www.plurk.com/samdhprint
https://angel.co/sam-designers-house
https://www.behance.net/samdhprint
https://visual.ly/users/samdesign
https://www.goodreads.com/user/show/82856669-samdhprint
https://www.f6s.com/samdhprint
https://codepen.io/samdhprint/
http://www.skem9.com/samdesign
http://www.letsliveexcitingly.co ... iew.php?postId=2390
http://magiadeamor.co/index.php/ ... to-digital-printing
http://www.franchipolis.com/samdesign
http://livestyle.community4um.de/u3524_samdesign.html
http://loknat.org/forum/member.php?action=profile&uid=5563
https://www.silvias-meerschweinc ... wprofile&u=6467
http://discutons.discuforum.info ... ewprofile&u=809
http://arshadeomran.ir/forum/member.php?u=32339
http://gsm-pager.ru/forum/profile.php?mode=viewprofile&u=216
http://board.space-wars.xooit.bi ... wprofile&u=1189
https://www.reddit.com/user/samd ... %DB%8C%D8%B3%D8%AA/
http://www.roozmenu.com/showthread.php?p=812573#post812573
http://samdhprint.blogfa.com/
http://samdhprint.bestblog.ir/
http://samdhprint.monoblog.ir/
http://samdhprint.jahanblog.net/
http://samdhprint.blogsky.com/
http://samdhprint.farsiblog.com/
http://www.samdhprint.loxblog.com/
http://samdhprint.azadblog.com/
http://samdhprint.parsiblog.com/
http://samdhprint.mihanblog.com/
http://samdesignershouse.mihanblog.com/
http://samdhprint.persianblog.ir/
http://samdhprint.roomfa.com/
http://samdhprint.tinablog.ir/
http://samdhprint.yektablog.net/
http://samdhprint.bigsite.ir/
http://samdhprint.parsablog.com/
http://samdhprint.takblog.net/
http://samdhprint.b88.ir/
http://samdhprint.novinblog.net/
http://samdhprint.ratablog.com/
http://samdhprint.blogiran.net/
http://samdhprint.tibablog.ir/
http://digitalprinting.bornablog.ir/
http://samdhprint.vistablog.ir/
http://samdhprint.sitearia.ir/
http://samgraphic.niloblog.com/
http://samdhprint.titablog.com/
http://samdhprint.aramblog.ir/
http://samdhprint.javanblog.ir/
http://samdhprint.lineblog.ir/
http://samdhprint.rozblog.com/
http://samdhprint.atizo.se/
http://samdhprint.aslblog.ir/
http://samdhprint.blogtez.com/
http://samdhprint.wordpressblog.ir/
http://digitalprinting.titrblog.ir/
http://samdhprint.7blog.ir
http://samdhprint.noblog.ir
http://samdhprint.madblog.ir
http://samdhprint.farazblog.com
http://samdhprint.deyblog.ir
http://samdhprint.blograz.ir/
http://bannerprinting.mihanblog.com/
http://samdhprint.bilog.ir/
http://samdhprint.blogia.ir/
http://chapgar.famblog.ir/
http://chapgar.wblog.xyz/
http://chapgar.blognini.ir/
http://chapgar.pasargadblog.ir/
http://chapgar.tadbirblog.ir/
http://chapgar.asalblog.ir/
http://chapgar.niliblog.ir/
http://chapgar.alvandblog.ir/
http://yaldaa.rayo.ir/
http://yaldaa.weblog.com.co/
http://chapgar.irancnblog.com/
http://chapgar.hamrahblog.com/
http://samdhprint.blog.ir/
http://chapgarsam.blogpart.ir/
http://yaldaa.blogr.ir/
http://yaldaa.kowsarblog.ir/
http://yaldaa.bipblog.ir/
http://yaldaa.kurdblog.ir/
http://yaldaa.ardabilblog.ir/
http://digitalprinting.fabino.ir/
http://digitalprintingsam.blogfa.com/
http://digitalprintingsam.bestblog.ir/
http://digitalprintingsam.monoblog.ir/
http://digitalprintingsam.jahanblog.net/
http://digitalprintingsam.blogsky.com/
http://bannerprinting.farsiblog.com/
http://bannerprinting.royablog.ir/
http://bannerprinting.loxblog.com/
http://bannerprinting.parsiblog.com/
https://bannerprinting.persianblog.ir/
http://bannerprinting.roomfa.com/
http://bannerprinting.yektablog.net/
http://bannerprinting.bigsite.ir/
http://bannerprinting.takblog.net/
http://bannerprinting.b88.ir/
http://bannerprinting.novinblog.net/
http://bannerprinting.blogparsi.com/
http://bannerprinting.blogiran.net/
http://bannerprinting.ratablog.com/
http://bannerprinting.vistablog.ir/
http://bannerprinting.tibablog.ir/
http://bannerprinting.bornablog.ir/
http://bannerprinting.niloblog.com/
http://bannerprinting.fardblog.com/
http://bannerprinting.sitearia.ir/
http://bannerprinting.rahatblog.ir/
http://bannerprinting.blogtez.com/
http://bannerprinting.aslblog.ir/
http://bannerprinting.atizo.se/
http://bannerprinting.lineblog.ir/
http://bannerprinting.javanblog.ir/
http://bannerprinting.aramblog.ir/
http://bannerprinting.titablog.com/
http://bannerprinting.vcp.ir/
http://bannerprinting.titrblog.ir/
http://bannerprinting.najiblog.com/
http://bannerprinting.7blog.ir/
http://bannerprinting.noblog.ir/
http://bannerprinting.madblog.ir/
http://bannerprinting.farazblog.com/
http://bannerprinting.deyblog.ir/
http://bannerprinting.blograz.ir/
http://bannerprinting.mihanblog.com/
http://banner.asalblog.ir/
http://banner.tadbirblog.ir/
http://banner.pasargadblog.ir/
http://banner.blognini.ir/
http://banner.wblog.xyz/
http://banner.famblog.ir/
http://banner.niliblog.ir/
http://banner.irancnblog.com/
http://businesscardsam.blogfa.com
http://businesscardsam.bestblog.ir/
http://businesscardsam.jahanblog.net/
http://businesscardsam.monoblog.ir/
http://businesscardsam.blogsky.com/
http://businesscard.farsiblog.com/
http://www.businesscard.loxblog.com/
http://businesscard.royablog.ir/
https://businesscard.persianblog.ir/
http://businesscard.roomfa.com/
http://businesscard.yektablog.net/
http://businesscard.bigsite.ir/
http://businesscard.takblog.net/
http://businesscard.b88.ir/
http://businesscard.novinblog.net/
http://businesscard.blogiran.net/
http://businesscard.niloblog.com/
http://businesscard.vistablog.ir/
http://businesscard.blogparsi.com/
http://businesscard.ratablog.com/
http://businesscard.bornablog.ir/
http://businesscard.rahatblog.ir/
http://businesscard.titablog.com/
http://businesscard.aramblog.ir/
http://businesscard.javanblog.ir
http://businesscard.lineblog.ir/
http://businesscard.atizo.se/
http://businesscard.aslblog.ir/
http://businesscard.blogtez.com/
http://businesscard.titrblog.ir/
http://businesscard.najiblog.com/
http://businesscard.blograz.ir/
http://businesscard.deyblog.ir/
http://businesscard.farazblog.com/
http://businesscard.madblog.ir/
http://businesscard.noblog.ir/
http://businesscard.7blog.ir/
http://sambusinesscard.mihanblog.com/
http://businesscard.asalblog.ir/
http://businesscard.tadbirblog.ir/
http://businesscard.pasargadblog.ir/
http://businesscard.blognini.ir/
http://businesscard.wblog.xyz/
http://businesscard.famblog.ir/
http://businesscard.niliblog.ir/
http://businesscard.irancnblog.com/
http://photoprint.asalblog.ir/
http://photoprint.tadbirblog.ir/
http://photoprint.pasargadblog.ir/
http://photoprint.blognini.ir/
http://photoprint.wblog.xyz/
http://photoprint.famblog.ir/
http://photoprint.niliblog.ir/
http://photoprint.irancnblog.com/
http://clothprintingsam.blogfa.com
http://clothprintingsam.bestblog.ir/
http://clothprintingsam.jahanblog.net/
http://clothprintingsam.monoblog.ir/
http://www.clothprintingsam.loxblog.com/
http://clothprintingsam.royablog.ir/
https://clothprinting.persianblog.ir/
http://clothprinting.roomfa.com/
http://clothprinting.bigsite.ir/
http://clothprinting.b88.ir/
http://clothprinting.niloblog.com/
http://clothprinting.bornablog.ir/
http://clothprinting.vistablog.ir/
http://clothprinting.blogparsi.com/
http://clothprinting.ratablog.com/
http://clothprinting.rahatblog.ir/
http://clothprinting.aramblog.ir/
http://clothprinting.javanblog.ir
http://clothprinting.lineblog.ir/
http://clothprinting.atizo.se/
http://clothprinting.aslblog.ir/
http://clothprinting.blogtez.com/
http://clothprinting.titablog.com/
http://clothprinting.titrblog.ir/
http://clothprinting.7blog.ir/
http://clothprinting.noblog.ir/
http://clothprinting.madblog.ir/
http://clothprinting.farazblog.com/
http://clothprinting.deyblog.ir/
http://clothprinting.blograz.ir/
http://clothprinting.mihanblog.com/
http://clothprinting.irancnblog.com/
http://www.joomi.ir/forum/forum100/thread3217/#post8487
http://eye4you.ir/forum/showthre ... C%D8%B3%D8%AA%D8%9F
http://forum.graphiran.com/thread10311.html#post63632
http://forum.kaghaz-rangi.com/fo ... ing/2449#topic_2449
https://www.ted.com/profiles/11123109
https://www.zotero.org/samdhprint
http://n4g.com/user/score/samdesign
https://www.diigo.com/user/samdhprint/b/477494351
https://getsatisfaction.com/people/samdhprint

You have to log in before you can reply Login | Sign Up

Points Rules