Recent Changes for "Export" - Sycamore Wiki

Export

PhilipNeustrom — 2007-12-23 11:30:20

brain dump added

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 97:	Line 97:
	+ + = Wiki Spot implementation brain dump = + * takes into account page security: user1, user2. + * user1 and user2 => set of groups they belong to + * set of groups you're in determines what you can do. + f(u) = (g1, g2, ..) + but what if i'm in g2 that differs only in editability but not readability? - we can only tell at global level..say for a page: + g1 edit, read g2 read g3 nada => i can edit, read + + if i am in a special group, say Events, then my dump is generated special-like...ok, let's just do that. + + dumps unique based on groups you're a member + + * where should the dump go? into the DB? + * dump to file then insert the file into the db..how do we do this without eating gigs of ram? -- can't even store objects that large...ok, so no db + * store something in the db -- unique_id, generated_time, file_url + * general get_file_url().. + * pulls from a config option for where big files are stored. + * new config option: machine domain name. can differ from wiki domain. each machine generating exports must have unique domain. + * we serve up by just sucking from that file on disk -- we assume it is locally accessible + * expire_exports() -- rm them.

Export

PhilipNeustrom — 2007-12-17 02:05:28

linked backup

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
- Sycamore currently has no export function. This feature is being explored.	+ Sycamore has a command-line export function. It's in {{{Sycamore/}}}. Just run {{{python export.py}}}.
Line 3:	Line 3:
- Current efforts are looking at determining the best format to use as a base or adopt for the Sycamore export format. - - - === Related Reading === - - [http://meta.wikimedia.org/wiki/Help:Export Export at MediaWiki] - [http://meta.wikimedia.org/wiki/DocBook_XML_export DocBook XML export at MediaWiki] - - - === How Others are Doing It === - - MediaWiki handles a few different export versions. There is XML export on page-by-page basis using a special URL (E.g {{{wikipedia.org/wiki/Special:Export/Sycamore}}}). The page-by-page export is only a current snapshot and does not include all revisions. - - Sample single page XML export from MediaWiki: [http://en.wikipedia.org/wiki/Special:Export/Sycamore Sycamore on Wikipedia (XML version)]. - - - There is an export that is a current snapshot of the entire site (does not include all revisions.). - - - There is an export that is a current snapshot of the entire site (does include all revisions.). - - == Discussion == - - A function to export/dump the contents of the wiki would be nice for backuping up the site, making it portable, or distributing it in another format (E.g HTML, DocBook or PDF) or any other desired format (based on a export theme or template). Have you done any work in this area? --["AdamDewitz"] - - This should be a priority, but I haven't done any work on this. A common export format (XML for human readability) and an import based on that format would be very cool. We could include all the non-temporary wiki data except user data (we have to include user name to user id information, though). We could then write separate utilities to use the wiki parser to process this XML format. We should have a 'full' export and a 'just the current data w/o users' export. --["PhilipNeustrom"] - - - I see that MoinMoin has a MoinDump utility. I wonder if some of the code could be used a base. Two possible export formats are Atom and DocBook. Both are XML-based and have good documentation. --["AdamDewitz"] - - That utility renders each page (static HTML). What would be more flexible would be dumping to our own XML format containing the raw data then writing something to convert our format to static HTML. --["PhilipNeustrom"] - - I haven't had time to look at the utility, but I read that it uses a hardcoded HTML template. If this is the case, the HTML-stuff in the template could be rewritten to output XML instead. - - Regarding the XML format, do want to create some Sycamore-specific XML format? I'm thinking its better to adopt a well defined format and if needed, extend it with our own namespace. --["AdamDewitz"] - - My understanding (I'm not an XML guru) is that because we have unique collection of data, that our representation would be pretty unique (we make something up). We have pages, which have versions, editors, edit dates, edit comments..files with the same, users with some attributes, etc. The 'page' content might be well served by something like DocBook, but there is a lot of other important data in the wiki install that a proper dump should take into account? --["PhilipNeustrom"] - I see two distinct uses for exporting content. - 1: Export data for backup. This needs to include all data (history, revisions, etc) and metadata (who did what with whom). - 2: Export data for other applications (static mirrors, prints (E.g books), etc.) This data would only have to be the current snapshot and not include all the data and metadata. - - Case 1 is more important that the 2nd. The data in the second can easily be derived from the output in case 1. If the export system was template-based, individual export needs could be met by modifying or creating a custom template. - - It looks like there has been some movement into using Atom for wiki content: - [http://www.intertwingly.net/wiki/pie/AtomForWikis AtomForWikis] - [http://www.xml.com/pub/a/2004/04/14/atomwiki.html An Atom-Powered Wiki] --["AdamDewitz"] - - Adam, do you want to try and tackle the export (#1) using Atom and see how it goes? I'd probably just look at the DB schema (buildDB.py). "Temporary" data shouldn't be bothered with for now -- this includes imageCaptions, thumbnails, cachedText/cachedTime, as well as links. We can move links into the export, as well as imageCaptions, later? --["PhilipNeustrom"] - - Yeah, I'll start looking into it. --["AdamDewitz"] - - It seems like the Atom work is being done in the area of the using the Atom API with wikis. I'm taking a look at what some of the other wiki projects use.	+ This is a per-wiki XML dump. This is different from a ["Backup" database backup]. The wiki export should contain all material needed to reconstruct the wiki in its entirety.
Line 144:	Line 93:
	+ + = Future = + * Limit to current versions of pages and files only + * Speed up utility (!!) and make it web-based

Export

PhilipNeustrom — 2007-12-17 01:50:38

initial export format finalized

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 60:	Line 60:
	+ Export a wiki. Note that some sections will appear in the export + only if you have permission. The security section, for instance, + will appear only if you are an administrator of the wiki in question. + + Export format looks something like: +

Export

PhilipNeustrom — 2007-12-16 01:29:06

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 57:	Line 57:
- Here's a sketch of the initial export format:	+ Here's the initial export format (Sycamore 0.1d)
Line 62:	Line 62:
- is_disabled="0 / 1" sitename="long site name">	+ is_disabled="True / False" sitename="long site name">
Line 64:	Line 64:
	+ <pages>
Line 65:	Line 66:
- <text> - current page text - </text>	+ <text>current page text</text>
Line 71:	Line 70:
- <text> - this version page text - </text>	+ <text>this version page text</text>
Line 76:	Line 73:
	+ </pages>
Line 78:	Line 76:
- <current> - <file name="filename" attached_to_pagename="page name"	+ <file name="filename" attached_to_pagename="page name">
Line 82:	Line 79:
- attached_to_pagename_propercased="Page Name"> - file content - </file> - </current> - <old> - <file name="filename" attached_to_pagename="page name"	+ attached_to_pagename_propercased="Page Name" + deleted="False">base64 encoded file content</file> + <file name="filename" attached_to_pagename="page name">
Line 92:	Line 86:
- attached_to_pagename_propercased="Page Name"> - file content - </file> - </old>	+ attached_to_pagename_propercased="Page Name" + deleted="True">base 64 encoded file content</file> + ...
Line 101:	Line 94:
- posted_time="unix tm"> - event text - </event>	+ posted_time="unix tm">event text</event>
Line 108:	Line 99:
- <acl pagename="page name" groupname="group name" may_read="0 / 1" - may_edit="0 / 1" may_delete="0 / 1" may_admin="0 / 1" />	+ <acl pagename="page name" groupname="group name" + may_read="True / False" may_edit="True / False" + may_delete="True / False" may_admin="True / False" /> + ...
Line 113:	Line 106:
	+ <defaults may_read="True / False" may_edit="True / False" + may_delete="True / False" may_admin="True / False" />
Line 142:	Line 137:
-

Export

PhilipNeustrom — 2007-12-11 04:37:05

what im doing right onw

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 55:	Line 55:
	+ + = Initial format = + Here's a sketch of the initial export format: + + {{{ + <sycamore version="0.1d"> + <wiki name="shortname" id="wiki_id" domain="domain name" + is_disabled="0 / 1" sitename="long site name"> + <settings key="value" ../> + <page name="page name" propercased_name="Page Name"> + <text> + current page text + </text> + <version propercased_name="Page Name!" edit_time="unix tm" + user_edited="username" edit_type="an enum edit type str" + comment="edit comment" user_ip="ip of edit"> + <text> + this version page text + </text> + </version> + </page> + + <files> + <current> + <file name="filename" attached_to_pagename="page name" + uploaded_time="unix tm" uploaded_by="username" + uploaded_by_ip="ip addr" + attached_to_pagename_propercased="Page Name"> + file content + </file> + </current> + <old> + <file name="filename" attached_to_pagename="page name" + uploaded_time="unix tm" uploaded_by="username" + uploaded_by_ip="ip addr" + deleted_time="unix tm" deleted_by="username" + deleted_by_ip="ip addr" + attached_to_pagename_propercased="Page Name"> + file content + </file> + </old> + </files> + + <events> + <event event_time="unix tm" posted_by="username" location="text here" + name="name of event" posted_by_ip="ip addr" + posted_time="unix tm"> + event text + </event> + </events> + + <security> + <acls> + <acl pagename="page name" groupname="group name" may_read="0 / 1" + may_edit="0 / 1" may_delete="0 / 1" may_admin="0 / 1" /> + </acls> + <groups> + <group name="group name"> + <user="user name"/> + <user="user name2"/> + <user="ip addr" type="IP"> + .. + </group> + </groups> + </security> + + <map> + <current> + <point pagename="page name" x="x loc" y="y loc" + created_time="unix tm" created_by="user name" + created_by_ip="ip addr" pagename_propercased="page Name" + address="address string" /> + ... + </current> + <old> + <point pagename="page name" x="x loc" y="y loc" + created_time="unix tm" created_by="user name" + created_by_ip="ip addr" + deleted_time="unix tm" deleted_by="username" + deleted_by_ip="ip addr" + pagename_propercased="page Name" + address="address string" /> + ... + </old> + </map> + </wiki> + </sycamore> + + }}}

Export

AdamDewitz — 2006-09-28 18:55:52

+ mediawiki stuff

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 6:	Line 6:
- Related Reading	+ === Related Reading ===
Line 10:	Line 10:
	+ + + === How Others are Doing It === + + MediaWiki handles a few different export versions. There is XML export on page-by-page basis using a special URL (E.g {{{wikipedia.org/wiki/Special:Export/Sycamore}}}). The page-by-page export is only a current snapshot and does not include all revisions. + + Sample single page XML export from MediaWiki: [http://en.wikipedia.org/wiki/Special:Export/Sycamore Sycamore on Wikipedia (XML version)]. + + + There is an export that is a current snapshot of the entire site (does not include all revisions.). + + + There is an export that is a current snapshot of the entire site (does include all revisions.).

Export

AdamDewitz — 2006-09-28 18:29:11

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 5:	Line 5:
	+ + Related Reading + + [http://meta.wikimedia.org/wiki/Help:Export Export at MediaWiki] + [http://meta.wikimedia.org/wiki/DocBook_XML_export DocBook XML export at MediaWiki]
Line 35:	Line 40:
	+ + It seems like the Atom work is being done in the area of the using the Atom API with wikis. I'm taking a look at what some of the other wiki projects use.

Export

AdamDewitz — 2006-09-28 18:25:17

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 3:	Line 3:
- Current efforts are looking at the [http://www.atomenabled.org/ Atom format] as a base for the Sycamore export format. - - Related links - - [http://www.atomenabled.org/ Atom format at atomenabled.org] - [http://www.intertwingly.net/wiki/pie/AtomForWikis AtomForWikis] - [http://www.xml.com/pub/a/2004/04/14/atomwiki.html An Atom-Powered Wiki]	+ Current efforts are looking at determining the best format to use as a base or adopt for the Sycamore export format.

Export

AdamDewitz — 2006-09-28 18:14:29

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 3:	Line 3:
	+ Current efforts are looking at the [http://www.atomenabled.org/ Atom format] as a base for the Sycamore export format.
Line 4:	Line 5:
	+ Related links
Line 5:	Line 7:
-	+ [http://www.atomenabled.org/ Atom format at atomenabled.org] + [http://www.intertwingly.net/wiki/pie/AtomForWikis AtomForWikis] + [http://www.xml.com/pub/a/2004/04/14/atomwiki.html An Atom-Powered Wiki]

Export

AdamDewitz — 2006-09-28 18:08:16

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 10:	Line 10:
- A function to export/dump the contents of the wiki would be nice for backup up the site, making it portable, or distributing it in another format (E.g DocBook or PDF) or any other desired format (based on a export theme or template). Have you done any work in this area? --["AdamDewitz"]	+ A function to export/dump the contents of the wiki would be nice for backuping up the site, making it portable, or distributing it in another format (E.g HTML, DocBook or PDF) or any other desired format (based on a export theme or template). Have you done any work in this area? --["AdamDewitz"]

Export

AdamDewitz — 2006-09-28 15:34:42

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 36:	Line 36:
- Yeah, I start looking into it. --["AdamDewitz"]	+ Yeah, I'll start looking into it. --["AdamDewitz"]

Export

AdamDewitz — 2006-09-28 02:38:14

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 35:	Line 35:
	+ + Yeah, I start looking into it. --["AdamDewitz"]

Export

AdamDewitz — 2006-09-28 02:37:08

Created page with content from Feature_Requests page

Differences for Export

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
	+ Sycamore currently has no export function. This feature is being explored. + + + + + + + == Discussion == + + A function to export/dump the contents of the wiki would be nice for backup up the site, making it portable, or distributing it in another format (E.g DocBook or PDF) or any other desired format (based on a export theme or template). Have you done any work in this area? --["AdamDewitz"] + + This should be a priority, but I haven't done any work on this. A common export format (XML for human readability) and an import based on that format would be very cool. We could include all the non-temporary wiki data except user data (we have to include user name to user id information, though). We could then write separate utilities to use the wiki parser to process this XML format. We should have a 'full' export and a 'just the current data w/o users' export. --["PhilipNeustrom"] + + + I see that MoinMoin has a MoinDump utility. I wonder if some of the code could be used a base. Two possible export formats are Atom and DocBook. Both are XML-based and have good documentation. --["AdamDewitz"] + + That utility renders each page (static HTML). What would be more flexible would be dumping to our own XML format containing the raw data then writing something to convert our format to static HTML. --["PhilipNeustrom"] + + I haven't had time to look at the utility, but I read that it uses a hardcoded HTML template. If this is the case, the HTML-stuff in the template could be rewritten to output XML instead. + + Regarding the XML format, do want to create some Sycamore-specific XML format? I'm thinking its better to adopt a well defined format and if needed, extend it with our own namespace. --["AdamDewitz"] + + My understanding (I'm not an XML guru) is that because we have unique collection of data, that our representation would be pretty unique (we make something up). We have pages, which have versions, editors, edit dates, edit comments..files with the same, users with some attributes, etc. The 'page' content might be well served by something like DocBook, but there is a lot of other important data in the wiki install that a proper dump should take into account? --["PhilipNeustrom"] + I see two distinct uses for exporting content. + 1: Export data for backup. This needs to include all data (history, revisions, etc) and metadata (who did what with whom). + 2: Export data for other applications (static mirrors, prints (E.g books), etc.) This data would only have to be the current snapshot and not include all the data and metadata. + + Case 1 is more important that the 2nd. The data in the second can easily be derived from the output in case 1. If the export system was template-based, individual export needs could be met by modifying or creating a custom template. + + It looks like there has been some movement into using Atom for wiki content: + [http://www.intertwingly.net/wiki/pie/AtomForWikis AtomForWikis] + [http://www.xml.com/pub/a/2004/04/14/atomwiki.html An Atom-Powered Wiki] --["AdamDewitz"] + + Adam, do you want to try and tackle the export (#1) using Atom and see how it goes? I'd probably just look at the DB schema (buildDB.py). "Temporary" data shouldn't be bothered with for now -- this includes imageCaptions, thumbnails, cachedText/cachedTime, as well as links. We can move links into the export, as well as imageCaptions, later? --["PhilipNeustrom"]